Tuesday, May 24, 2011

Java Application Server Performance Tuning (i.e. JBoss, Weblogic, Apache, Tomcat)

There are many Java application running on top of JBoss, Weblogic, Apache, Tomcat, etc. But at the end, a lot of people looking for performance tuning and there is one key area we need to take note. To minimize Java out of memory issue, better way to manage GC garbage collection, adjust needed Java JVM run time memory based on hardware physical memory, operating system, etc. Below some sharing is based on one of the e-Commerce project that I tuned on the Java application server before, and so far it getting better, stable and improve a lot without downtime for few weeks. It able to handle more than 5000 hits PV at same time without giving us any worry.

This is based on Redhat Linux Ent 5 64bit, 2 x CPU, 32GB Memory environment.There are similar way if you want to change in Microsoft Windows Server, but some changes in registry end. I won't explain too much here, as you can find those info easily from Internet. Below just a reference for those people spend a lot of time to look for Java JVM or JBoss or Weblogic or Apache or Tomcat application server tuning, and please adjust accordingly based on your environment and requirement from hardware, software, application server, operating systems, etc. Note: Most important thing is you need to finetune your coding, as below won't make your application fly... :)

1. Change /etc/security/limits.conf

* - nofile 65536
root soft nofile 4096
root hard nofile 65536

If you want to change it temporary, then you can use below command:-
ulimit -Hn 65536
ulimit -Sn 4096

2. Change /proc/sys/net/ipv4/ip_local_port_range

1024 65535

3. Change /etc/sysctl.conf

kernel.shmall = 4294967296
kernel.shmmax = 4294967296 (68719476736 if more than 4G memory)
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 65536
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_max = 4194304

net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.core.netdev_max_backlog =8096
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_window_scaling = 0

You can activate the changes immediately using below command:-
/sbin/sysctl -p
/etc/rc.d/init.d/network restart
4. Java JVM run.conf tuning changes (if physical memory less than 4GB, then better set to 3072; in my environment, i am using 24GB setting)

JRockit JDK 64bit (Realtime)
-Xms8192m -Xmx8192m -Xns512m -XgcPrio:deterministic -XpauseTarget=30ms -XXnosystemgc -Xgc:singlecon -XXlazyUnlocking -XXcallProfiling

JRockit JDK 64bit (Other)
-Xms8192m -Xmx8192m -Xgc:gencon -XXnosystemgc -XXtlasize:min=3k -XXkeeparearatio=0 -Xns:48m -XXlazyUnlocking -XXcallProfiling -XlargePages -XXcompactRatio:1

Other JDK 64bit
-server -Xms8192m -Xmx8192m -Xmn512m -XX:PermSize=512m -XX:MaxPermSize=512m -Xss256k -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:GCTimeRatio=19 -Xnoclassgc -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime

5. Change on database connection (if using Oracle - oracle-ds.xml) (remember to put in < & >)





check-valid-connection-sql>select 1 from dualsql

6. Change on Java application server http connector (i.e. server.xml) (remember to put in < & >)

Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}"
maxThreads="1024" strategy="ms" maxHttpHeaderSize="8192" minSpareThreads="100" maxSpareThreads="250"
emptySessionPath="true" maxKeepAliveRequests="1" enableLookups="false" acceptCount="2048" connectionTimeout="10000" disableUploadTimeout="true" redirectPort="8443" URIEncoding="UTF-8" compression="1" compressableMimeType="text/html,text/xml,text/css,text/javascript, application/x-javascript,application/javascript"
There are some reference on this setting, and hope it help.

Memory     Connections     maxThreads     acceptCount
24G        above 1500      4096~8192      4096~8192
16G        1200~1500       2048           2048

12G        1000~1200       1526           2048
8G         300~1000        1024           2048       
4G         below 300       512            1024
2G         below 100       256            512

7. Change on httpd.conf if using other Web server i.e. Apache

<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 15
ServerLimit 2000
MaxClients 1000
MaxRequestsPerChild 10000
<IfModule worker.c>
StartServers 3
MaxClients 2000
ServerLimit 25
MinSpareThreads 50
MaxSpareThreads 200
ThreadsPerChild 100
MaxRequestsPerChild 0

So, above Java tuning setting been tested before, and hope it help on your research or testing environment and even run in production once you tested ok. Below some checking command help you to see some info.

To check the connections:-

netstat -ant | grep 8080 | wc -l
netstat -ant | grep 8080 | awk '{print $6}' | sort | uniq -c | sort -n

ps -ef | grep java (to find the id, i.e. 10122)
jstack 10122
jstat -gcutil 10122 1000 or jstat -gc 10122 1000

Never Try! Never Know! Just Try This Java Tuning Setting!