Java Application Server Performance Tuning (i.e. JBoss, Weblogic, Apache, Tomcat)
There are many Java application running on top of JBoss, Weblogic, Apache, Tomcat, etc. But at the end, a lot of people looking for performance tuning and there is one key area we need to take note. To minimize Java out of memory issue, better way to manage GC garbage collection, adjust needed Java JVM run time memory based on hardware physical memory, operating system, etc. Below some sharing is based on one of the e-Commerce project that I tuned on the Java application server before, and so far it getting better, stable and improve a lot without downtime for few weeks. It able to handle more than 5000 hits PV at same time without giving us any worry.
This is based on Redhat Linux Ent 5 64bit, 2 x CPU, 32GB Memory environment.There are similar way if you want to change in Microsoft Windows Server, but some changes in registry end. I won't explain too much here, as you can find those info easily from Internet. Below just a reference for those people spend a lot of time to look for Java JVM or JBoss or Weblogic or Apache or Tomcat application server tuning, and please adjust accordingly based on your environment and requirement from hardware, software, application server, operating systems, etc. Note: Most important thing is you need to finetune your coding, as below won't make your application fly... :)
1. Change /etc/security/limits.conf
* - nofile 65536
root soft nofile 4096
root hard nofile 65536
If you want to change it temporary, then you can use below command:-
ulimit -Hn 65536
ulimit -Sn 4096
2. Change /proc/sys/net/ipv4/ip_local_port_range
1024 65535
3. Change /etc/sysctl.conf
kernel.shmall = 4294967296
kernel.shmmax = 4294967296 (68719476736 if more than 4G memory)
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 65536
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_max = 4194304
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes=2
net.ipv4.tcp_keepalive_intvl=2
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.core.netdev_max_backlog =8096
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_window_scaling = 0
You can activate the changes immediately using below command:-
/sbin/sysctl -p
/etc/rc.d/init.d/network restart
4. Java JVM run.conf tuning changes (if physical memory less than 4GB, then better set to 3072; in my environment, i am using 24GB setting)
JRockit JDK 64bit (Realtime)
-Xms8192m -Xmx8192m -Xns512m -XgcPrio:deterministic -XpauseTarget=30ms -XXnosystemgc -Xgc:singlecon -XXlazyUnlocking -XXcallProfiling
JRockit JDK 64bit (Other)
-Xms8192m -Xmx8192m -Xgc:gencon -XXnosystemgc -XXtlasize:min=3k -XXkeeparearatio=0 -Xns:48m -XXlazyUnlocking -XXcallProfiling -XlargePages -XXcompactRatio:1
Other JDK 64bit
-server -Xms8192m -Xmx8192m -Xmn512m -XX:PermSize=512m -XX:MaxPermSize=512m -Xss256k -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:GCTimeRatio=19 -Xnoclassgc -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
5. Change on database connection (if using Oracle - oracle-ds.xml) (remember to put in < & >)
min-pool-size>20
max-pool-size>200
blocking-timeout-millis>50000millis
idel-timeout-minutes>10idel-timeout-minutes
check-valid-connection-sql>select 1 from dualsql
6. Change on Java application server http connector (i.e. server.xml) (remember to put in < & >)
Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}"
maxThreads="1024" strategy="ms" maxHttpHeaderSize="8192" minSpareThreads="100" maxSpareThreads="250"
emptySessionPath="true" maxKeepAliveRequests="1" enableLookups="false" acceptCount="2048" connectionTimeout="10000" disableUploadTimeout="true" redirectPort="8443" URIEncoding="UTF-8" compression="1" compressableMimeType="text/html,text/xml,text/css,text/javascript, application/x-javascript,application/javascript"
There are some reference on this setting, and hope it help.
Memory Connections maxThreads acceptCount
24G above 1500 4096~8192 4096~8192
16G 1200~1500 2048 2048
12G 1000~1200 1526 2048
8G 300~1000 1024 2048
4G below 300 512 1024
2G below 100 256 512
7. Change on httpd.conf if using other Web server i.e. Apache
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 15
ServerLimit 2000
MaxClients 1000
MaxRequestsPerChild 10000
IfModule>
<IfModule worker.c>
StartServers 3
MaxClients 2000
ServerLimit 25
MinSpareThreads 50
MaxSpareThreads 200
ThreadsPerChild 100
MaxRequestsPerChild 0
IfModule>
So, above Java tuning setting been tested before, and hope it help on your research or testing environment and even run in production once you tested ok. Below some checking command help you to see some info.
To check the connections:-
netstat -ant | grep 8080 | wc -l
netstat -ant | grep 8080 | awk '{print $6}' | sort | uniq -c | sort -n
ps -ef | grep java (to find the id, i.e. 10122)
jstack 10122
jstat -gcutil 10122 1000 or jstat -gc 10122 1000
