Snort High CPU Usage and Blank Intrusion Detection Screen

[Guest]

[gdPAC]

This week, we began experiencing VOIP quality issues.  Investigation revealed the snort process on our Endian 2.2 firewall is using 67-99% CPU.  I manually updated snort rules earlier this week and the problem surfaced after that.  I am unable to view the Intrusion Detection settings in the Endian GUI because the screen comes up blank.  If I kill the snort process in an SSH session, all traffic stops until the firewall is restarted.  Does anyone have a suggestion for fixing this before I do a fresh reinstall over the weekend?

Thanks.
Glen

[mrkroket]

Maybe its related to: http://efwsupport.com/index.php?topic=947.0

From console try to restart snort with debug log:

restartsnort.py -d -f

check that there isnt any error.  You should see something like:


......
......
.......
2009-10-23 15:43:49,311 - restartsnort.py/enabled_rule_targets[20065] - DEBUG - Stop snort
snort (pid 19990) is running...
Stopping snort:                                            [  OK  ]
snort is stopped
2009-10-23 15:43:49,576 - restartsnort.py/enabled_rule_targets[20065] - DEBUG - Start snort
2009-10-23 15:43:49,582 - restartsnort.py/enabled_rule_targets[20065] - INFO - Starting SNORT...
Starting snort:                                            [  OK ]


If snort says failed, there was a problem with some updated rule.
To see what program eats up the CPU % use console command top

[gdPAC]

Snort restarted without errors.  I've been monitoring with TOP most of the day.  Snort has been using >67% CPU most of the time and seems to be linked to the quantity of VOIP traffic.

top - 17:06:27 up  1:50,  1 user,  load average: 0.35, 0.66, 0.59
Tasks:  64 total,   3 running,  61 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.0%us,  1.0%sy,  0.0%ni, 70.7%id,  0.0%wa,  0.7%hi,  2.7%si,  0.0%st
Mem:    449780k total,   338476k used,   111304k free,    37088k buffers
Swap:   907664k total,        0k used,   907664k free,    84616k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10760 root      15   0 82804  59m 1336 R 27.9 13.6   0:57.87 snort
 3891 openvpn   15   0  4332 2480 1292 S  0.3  0.6   2:46.19 openvpn
11005 root      15   0  1976 1004  788 R  0.3  0.2   0:00.10 top
    1 root      15   0  1504  552  468 S  0.0  0.1   0:00.44 init

The Services > Intrusion Detection configuration screen is still blank after restarting snort.

Thanks!
Glen

[mrkroket]

Did you try to disable the VOIP.rules on snort?
https://EFW:10443/manage/ips/rules/

If you can't enter GUI, use console to rename the voip rules so snort doesn't use them:
cd /etc/snort/rules/auto
mv  emerging-voip.rules emerging-voip.rules.out
restartsnort.py -d -f

If that doesn't help, the problem shouldn't be on voip rules.
You can try to stop the snort daemon and run it in debug mode (showing errors on console). I never did that so I can't help. try to kill snort and run it by using the command snort

[gdPAC]

The URL /manage/ips/rules/ doesn't work on my EFW.  404.  Should that work in v2.2?

I renamed the VOIP rules and restarted snort.  No change.  I stopped the process with killall and tried to restart with "snort" but it wanted parameters.  restartsnort.py -d -f worked, though.

I tested throughput at speedtest.net and monitored running EFW processes with TOP.  Snort consumed 99%+ CPU during the speedtest transfers.  So it looks like a general thoughput issue, not just VOIP traffic.

At this point, I am going to try to reset to factory default, then restore config from a backup.  If that doesn't work, then a fresh reinstall and reconfig.  Any other suggestions before I get started?

Thanks for the help.
Glen

[gdPAC]

Resetting to defaults and restoring from backup had no effect on snort's CPU-hogging ways.  While researching, I noticed 2.3 is due out the 27th.  So I renamed all the snort auto rules to *.rules.out and restarted snort.  It is now much more CPU-kind and red throughput is back to normal.  Phone voice quality sounds good, but I'm the only one using the phone right now.  It'll be Monday before it is back under full load.  The Intrusion Dectection setup page is still blank, so I'm unable to use the GUI to configure snort.  I'll just wait to upgrade to 2.3 next week and hope that fixes the problem.

Thanks again for the help.
Glen

[mrkroket]

Yes, the manage/ip URL is on EFW 2.3 only. It now has much better control on IDS.

You can change IDS setting (or any part of the GUI) via console commands, its not hard.
go to /var/efw
you'll see lots of dirs with settings files. Edit the files and restart the as.sociated script
I.e. for snort:
nano /var/efw/snort/settings
Edit the file options. 0 (or off) disable the selected option.
When you are done, Ctrl+O to save and Ctrl+X to quit
Restart the snort process:
restartsnort.py -d -f

If you are going to upgrade to 2.3, be warned. It's still a Release Candidate, with much better options but some little issues (nothing big).
Still, there are some issues also on IDS for EFW 2.3. There is a post in the forum related to the error (an incorrect rule that appears if you update snort rules).

[gdPAC]

So the announcement on the home page of Endian.com "2.3 Available from Oct. 27, 2009" isn't a stable release announcement?

My "solution" wasn't one.  When enough people used the phone, the jitter and dropouts returned.  Your suggestion to edit the settings file to disable Snort seems to have done the trick.  CPU usage is down and throughput is up.  But the Services > Intrusion Detection screen is still blank and will not allow me to configure Snort.  Something is broken and I have no idea how to fix it.

Thank you.
Glen

[mrkroket]

So the announcement on the home page of Endian.com "2.3 Available from Oct. 27, 2009" isn't a stable release announcement?

I was refering to 2.3 Release Candidate ( =1&cHash=23cfe22e7c]http://www.endian.com/es/compania/news/article/endian-firewall-community-23-release-candidate/?tx_ttnews[backPid]=1&cHash=23cfe22e7c), out from 17 Sep 09. If you use a lot of VOIP you'll need good QoS queues. 2.3 have an improved QoS. Do you maths and reserve enough bandwidth for your VOIP calls to reduce jitter and ensure voice quality.

[gdPAC]

Network QOS is configured and with Endian Traffic Shaping, VOIP was working flawlessly for months, even under high bandwidth usage.  Only when Snort started taking 67%+ CPU did we experience these problems.  Something is wrong with IDS in my current configuration and Snort is killing red interface throughput.  I am not aware of a way to reinstall just Snort in EFW 2.2.  If EFW 2.3 stable is being released this week, I'll cross my fingers and upgrade, hoping the problem will be resolved.  If not, I'll do a full reinstall.

Thank you.
Glen