I have a same problem on Endian 2.4.1 proxy. My current caching settings HDD : 2048MB, RAM : 64 MB, Min. Obj. Size : 0KB, Max. Obj. Size : 1024KB.. When the total memory usage(Used+buffered+cached) coming around %90 the internet speed extremely slow down!
I had to restart the server every morning because of this issue
My system is an HP PC with 4GB RAM and 120GB HDD..