EFW Support

Support => EFW SMTP, HTTP, SIP, FTP Proxy Support => Topic started by: crisman on Friday 08 May 2015, 01:27:01 am



Title: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Friday 08 May 2015, 01:27:01 am
Hi,

I'm using the EFW 3.0.5 Beta 1 on a HP DL380 G4 with 2 CPUS Xeon 3.6 Ghz DP and 6 Gb Ram and have been testing 3 Active Directory users and most of the times the Squid gets 100% CPU usage, I've already disabled caching, is there any fine tuning to fix this?

Thanks.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Friday 08 May 2015, 05:28:53 pm
ClamAV is running? If is running try to disable the AV.
Look at the web filtering. Delete all policies and create again one without av filtering. Restart.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Friday 08 May 2015, 06:02:57 pm
Hi,

Yes Clamav is running, I will try without Clamav but if this feature is present there should be a way to have both running without problem, this is a bug with Clamav?

Thanks.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Friday 08 May 2015, 08:00:13 pm
From my experience with Endian (not allot, but i have a machine in production from over 2 years now, 50 users) Clam AV with proxy have problems. In fact the bigest problems that i have with Endian 3.0 and now 3.0.5 is proxy with clamAV and second is web filtering (Endian has a default policy with AV filtering).
When i disabled clamAV the box was very stable with 3.0 and now with 3.0.5. I also deleted the default web policy and created a new one without AV filtering. I to have disabled cache from squid.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Friday 08 May 2015, 08:33:31 pm
I see!

But using Clamav would be more secure.
BTW what is your hardware?

Thanks.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Friday 08 May 2015, 09:31:25 pm
For the first machine with 3.0 it was a old Sempron 1.8Ghz single core, with 2GB RAM SDR and 80GB HDD Sata 1, 3 NIC's ( 1SiS, 1 Intel, 1 Realtek 1Gb)
Now i have a Pentium 2.8 Dual core, 4GB RAM, 250 GB HDD Sata2, 3 NIC's (2Realtek 1Gb, 1Intel).
ISP speed 50-60Mb down/20-30 Mb up

You are right, the ClamAV its good but under this circumstances better off.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Friday 08 May 2015, 11:09:46 pm
Are you using the Web Filter profiles?



Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Friday 08 May 2015, 11:44:18 pm
No. Only acces policy. I have 2 policies under ACCESS POLICY: One with  of sites that i block without a profile (just "acces denied") and the second one is "filter any to any" without blocking anything and without AV,  using the profile created under WEB FILTER without AV.
The Endian comes with a default acces policy "filter any to any" with a profile that has AV enabled.
I deleted that and recreat my policy from above with a new profile from web filter a profile without AV.
Acces policies is working but filtering with profiles dont work. I dont know why. I tried allot to make filtering work, but......


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Friday 08 May 2015, 11:59:43 pm
Ok,

Later I will try to add the profiles working.
At the moment, I only want the users from my AD to access the web without problems.
Do you use Endian for a long time or you are also new to EFW?
Do you know when a stable 3.0.5 will be released?
It seems this project is a little forgotten! No much community support, and if this has been around here for several years why not 64 bit version yet since all others already have 64 bits and also IPv6 support?

Thanks.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Saturday 09 May 2015, 12:35:41 am
Why did you integrated with AD......can you try without integration?
Disable clamAV and delete profiles and Acces Policies. You can create policies later. Restart and see if is working.
Im using Endian from over 2 year now ( i think ) and before ( for short time) other Distros, but Endian catch me. Version 3.0 is the first contact with it.
I dont know when is the next update. Anyway, im in proccess (4 months now) of returning to other Distros because lack of support in Community version, although the GUI is nice and intuitive. I like the configuration options, but not all are working OK.



Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Saturday 09 May 2015, 12:39:01 am
Keep me updated.
See you Monday.
Have a nice weekend.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Saturday 09 May 2015, 12:52:23 am
I have to integrate with AD because we have several users and only a few must have internet access and this way I can easily give permissions to whom have access or not.
I used Microsoft ISA Server but the support and product has been discontinued but it works well and fast.

At this time I having again Squid using most of the time all CPU resources, it seems it works fine if I restart the service but a few hours later its start consuming lost of CPU, very unstable and I'm only testing with five users but need to use it with 30 users!!!
Probably I will have to find another Proxy solution with AD integration.  :(

Have a nice weekend too!


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Monday 11 May 2015, 09:37:41 pm
Did you disable clamAV? Did you recreat acces policies again?


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Monday 11 May 2015, 10:50:13 pm
Hi,

Yes I've disabled all that options, so with only proxy and no Virus scanning and no Web Filtering I still have Squid very often taking all CPU resources.  >:(

Regards.



Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Tuesday 12 May 2015, 11:42:01 pm
The rules in Squid are all enabled or just log and pass?
Try disable all the rules and see what the CPU do?


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Wednesday 13 May 2015, 01:05:59 am
Hi,

At this time I will try out other solutions and it seems IPFire to be a good replacement.

Thanks.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Wednesday 13 May 2015, 06:05:48 pm
OK. I play a few days with IPFire, Is ok, smooth, everything work great but it doesnt have the potential of Endian.
Anyway, this year i want to change the Endian box myself. I will  try Psense. The GUI is not that intuitive, but the community support and updates are great.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Wednesday 13 May 2015, 06:21:43 pm
It doesn't seems that Endian as so much potential, it lacks 64 bit so cannot use all memory and other hardware  resources and it does not provide IPv6, I only would like to use it has a Proxy for my company, at home I use Sophos UTM Home license, Sophos has really lot of potential and much more feature rich than Endian and the support forum is awesome, I only cannot use it at work because the license does not permit.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: gsv on Friday 15 May 2015, 06:30:59 pm
I know it has allot of bugs, but for a free version in a company it has allot of features.......the only big problem i saw is the lack off support.
The 64 bit version  i dont considered a big problem because Endian can sustain a network with 50, 100, 200 user with a modest box. My old box for example it was a old system and it handled my users ok (40-50 users) with 2 subnets.
Anyway it has allot of bugs, thats why i decide to test other Distros and change in the far future Endian.
Good luck


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: mrkroket on Saturday 23 May 2015, 12:32:11 am
it has a kernel PAE, this means it can use more memory. I don't know if it's stock or not, but on previous versions I've installed it.
I have an Endian 2.4.0 with 4GB ram.

Disable antivirus, it's a resource hog. Also go to Services->Antivirus Engine and reduce all values to 1. I know it's a lack, but it's better than nothing.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Saturday 23 May 2015, 01:19:50 am
it has a kernel PAE, this means it can use more memory. I don't know if it's stock or not, but on previous versions I've installed it.
I have an Endian 2.4.0 with 4GB ram.

Disable antivirus, it's a resource hog. Also go to Services->Antivirus Engine and reduce all values to 1. I know it's a lack, but it's better than nothing.

Hi,

I'm using another different solution FW and it also uses antivirus and web filtering and I don't get the CPU at maximum, so it should be a Endian Bug?



Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: mrkroket on Saturday 23 May 2015, 04:01:45 am
What solution? I've been searching any opensource UTM solution, and no one comes closer to what Endian offers.

Probably yes, it must be an Endian bug. It has plenty of bugs, everywhere.
 But once you fine tune it, it's very stable.
I have a EFW with about 250 users, all but AV enabled (IDS, Content Filter, 200+ firewall rules, 8+ OpenVPN's connections, IPSec....), and I have like 25% CPU time (4 core).

You can't expect support from EFW community, it's a nice product but you are pretty much alone on it.
To fine tuning and add custom packets you must have some linux experience.
It's not a plug and play.




Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: crisman on Monday 25 May 2015, 11:59:02 am
Hi,

I've tried with IPfire and PFSense and both worked fine, you must also have some experience on linux/unix to make some fine tuning.
With Endian, when I turn the server on, it works fine but an hour later I get CPU stress by Squid, don't understand why?
You have a lot of users so I'm not sure if IPFire or PFSense will behave well but at least on PFSense the community helps a lot.


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: burja2 on Wednesday 10 June 2015, 01:59:59 pm
Hello Crisman,

DL380 with 2 Xeon and 6 GB...? that's still pretty decent stuff IMHO. You shouldn't have that kind of problems.

I'm with EFW since 2.2.1 (when they were truly opensource...) It's epic the proxy implementation in EFW had lots of problems, hard to tell if it really work or just simulating... anyway...

With 3.0.5, (clean install - not an upgraded version, done last Sunday) I'm now enjoying (at last) a http proxy that works !!  including AV and even SNORT ( with ALL its rules enabled with drop. All my FW rules (about 40...) are set to "allow with FS"). So, if you'd previously upgraded from 3.0 (or else) a clean install will clearly help. It solved many problems for me.

My two teens are giving EFW a real beating since Monday. Yet my CPU (AMD Athlon 6600 - old stuff and only 4GB DDR2) bumped tonight at a "whooping" 10% load after supper for a full 3h then went down as my kids went to bed. I'll continue to monitor CPU's behavior to see if it could  the roof like you.

Last thoughts, through my years of experimentation with EFW, I found the hdd being the weak point. I've gone through 3 drives since 2007/8. I'm currently with a small SSD drive (60Gb nothing "extravagant"). So, maybe a hard disk currently have a "near-dead" experience ;-)  and your CPUs are taking the told, or maybe a lack of swap space? (you should have 12Gb of swap space)

 for thoughts



Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: svritc_81 on Friday 21 August 2015, 04:47:57 pm
Can you try these settings in squid.conf

hosts_file /etc/hosts
dns_nameservers x.x.x.x x.x.x.x
cache_replacement_policy heap LFUDA
cache_swap_low 90
cache_swap_high 95
maximum_object_size_in_memory 50 KB
cache_dir aufs /var/spool/squid 40000 16 256
cache_mem 100 MB
logfile_rotate 10
memory_pools off
maximum_object_size 50 MB
quick_abort_min 0 KB
quick_abort_max 0 KB
log_icp_queries off
client_db off
buffered_logs on
half_closed_clients off


Title: Re: EFW 3.0.5 slow Proxy and Squid using all CPU
Post by: cocoalcazar on Friday 25 September 2015, 01:09:38 am
to put that configuration and change is deleted via web,  any other option? please.  Tghank you.