This looks good.
What is your network scenario? May be there is a large broadcast when number of user increases.
i have 2 servers, for entire net, 1 efw 2.4.1 with aprox 25 users, is an ibm, with 8 gb in ram, the other server is a centos 5, this have aprox 35 users, this have 1 gb in ram,