6 months in ING Direct Spain

Today marks 6 months of my tenure in ING Direct Spain. So it is helpful to have some reflections.
1. My job is primarily administrative. During the time, I was involved in a couple of troubleshooting issues, for which we are able to find out a root cause and proposed a conclusive solution.

2. The second thing is my effort to refactor our JBoss infrastructure and to use git to track and manage the change. One lesson I learned from that is not matter what sophisticated we are using, the key of success is ability a) to figure out what is specific and what is common and b) to reduce number of configuration items.
At the beginning I tried to track configuration items on each machine as one git branch but it turns out into a horror of hundreds of un-manageable branches. At the end I thrown it and start to divide the monolithic configuration into smaller parts, parametrize them for reuse. After that using git or any other VCS for tracking them is just as walk in a rose's garden.

3. The third thing I have started is documentation of our infrastructure, I decided to use wiki for doing it. It turned out to be a good choice. The document is very up to date and practical because I use the document during my daily work so whenever I find a gaps between the document and the reality, I correct it. Update documentation is not separated task but a part of my day-to-day activities.

4. An the last thing is my research to look for a central authentication solution for our unix farm of mix LINUX, AIX and Solaris. I have tested PAM & Open LDAP combination for a while but at the end I have selected and recommended Likewise with the existing Active Directory as authentication server due to a lower cost of installation and operation.

java.lang.OutOfMemoryError Out of swap space?

I have encountered this error in one of our Hotspot JVMs. Googling for a while without definitive result, I decided to look at source code of Hotspot VM (happily the source code is available in Sun website).
Within few minutes, with very little effort, I found that JVM print out this message when got NULL from calling malloc. So it turns out to find why malloc fails.
When OS does not have RAM to satisfy malloc, it may try to swap out unused real memory to disk and if the swap space is configured too small, the error appears. But this is just one cause.
Other more likely cause is that JVM run out of memory address space and this is our case. It can happen when we run 32 bit JVM and total memory used by the JVM exceeds magical limit of 3 GB (on 32 bit kernel LINUX).

Bringing more development practices to the world of administration

In tradition software development and system administration tends to be separated. So useful practices in software development are not known by many administrators. I would say that the world of system administrations lack very much behind software development's world.
While using Version Control System (VCS) is obvious thing in any software project, not many administrator use this very useful tool during their daily work. The result is a dirty environment where a configuration's file is backup manually by renaming before making change. Also there is difficult to perform a rollback and reason and time of change is not well maintained.
Other often well encouraging software principles are to achieve re-usability, readability, explicitness as well as to avoid duplication. On the other hand an administrator cares much less about them. I have seen an environment where there are hundreds of scripts poorly written, repetitive, difficult to understand and maintained. I do thing that by using technique as re-factoring the situation can be improved significally.

Articles about deployment and operation are rare

Deploy and run a software is a vital part of the full picture "make value from the software". However does not draw much attentions from community.I listed here links to few blogs and websites talking about this subject matter.
http://www.build-doctor.com
http://www.agileweboperations.com/
http://en.oreilly.com/velocity2009
http://reductivelabs.com/trac/puppet/wiki/FrequentlyAskedQuestions
http://ganglia.info/
http://www.cfengine.com/
http://trac.mcs.anl.gov/projects/bcfg2
http://automateit.org/
http://wiki.opscode.com/display/chef/Home

Joining to ING Direct Spain

ING Group decided not to launch ING Direct Operation in Japan, as result I moved to ING DIRECT Spain. As for non European citizen, the visa process take long time for me. But finally I landed to Spain Office at the end of July.
This is logical step in my career, I want to stay with ING DIRECT and the job market has still very tough. I returned to Europe after 12 years, I moved from Prague to Hanoi in July, 1997.
The first two weeks in the Spain Office is exciting, people are friendly, HR is supportive, office is nice, lunch is free, job is interesting. But like you got marry and went to honeymoon, this period does not last forever. Back to reality, I have to be pragmatic, find the way to integrate to the team and the community, be helpful and continue the journey in my career path.
Weather is Spain in this season is good especially in the morning, drop me a call or a message whenever you go Madrid, I would love to meet you for a drink, tapas.

My Resume - SUMMARY of SKILLS

---

System administration, support and maintenance

• Responsible for large scale WebSphere Application Server ND and WebSphere MQ infrastructure used by iFlex (Oracle Flexcube) retail banking system for a leading consumer bank.
• Led a team, that provide remote and onsite support 24/7 distributed critical J2EE application based on WebLogic and Oracle Database for immigration control.
• Directly involved in identification and analysis critical problems, was interface point with vendors, customers, and management in incident handling, problems solving process
• Well versed in using scripting language (RUBY, PYTHON) to automate all kinds of administrative activities including software installation, update; configuration management; application deployment, update; testing; monitoring; and log file management

Java and J2EE Application Server

• More than 5 years experience with JAVA platform and J2EE framework particularly Websphere, WebLogic. Directly involved in capacity planning, application packaging, deployment, monitoring, troubleshooting that includes sizing memory, thread pools, setting transaction's timeout, maintaining resource pool, etc..
• Well understood internal JVM and its assembler language (byte code). Hands-on experience in tools and technique for identifying JVM and J2EE problem such as analyzing heap dump, core dump, garbage collector, decompile byte code, injection tracing code into application and container code.

Messaging System

• More then 7 years in design and implementation of software systems that communicate with each other using asynchronous messages. Understand message creation and processing patterns (Enterprise Integration Patterns) and their applications in real world systems.
• Responsible for deployment IBM MQ as message backbone for many applications. The responsibility includes interfacing with development team for planning and configuration, testing, trial run, installation, monitoring and troubleshooting

Database System

• More then 10 years extensive working experience on RDBMS Oracle in both administrator and developer role. Obtained certification on Oracle DBA in 1999, hand-on experiences in large site capacity planning, performance tuning, high availability using Real Application Cluster (RAC) and Data Guard.
• Involved in database design for many enterprise projects, assisted implementation team in evolving database schema, data migration and cleansing.

Operating System and Hardware

• More than 10 years experience in design, installation, administration and troubleshooting complex IT infrastructure including UNIX clustering system, IBM FC and iSCSI SAN, CISCO router and PIX firewall
• Hand-on experience in AIX administration including HACMP. Obtained certification on AIX administration in 1999. Performed installation, configuration, tuning many AIX based systems(s), making them ready for deployment Oracle Database and J2EE application server.
• Involved in technical pre-sale activities including sizing a system, proposal preparation, technical presentation and alike.

Software Development

• More than 15 years experience in developing software products with deep knowledge of architecture / design of multi-tiered client-server systems based on middleware including Tuxedo, J2EE (WebSphere, WebLogic) and other emerging Web framework as Ruby on Rails.
• Skillful in many programming languages including C/C++, JAVA, SQL, PL/SQL, RUBY. Passionate for the simple and easy to understand software with rich domain model. Well versed in OOP, design patterns (Design Patterns - GoF, Pattern for Enterprise Application Architecture - PEAA, Domain Driven Design).
• Strong knowledge of algorithms and data structures and their applications in writing a high performance, time critical system for complex business domain.
• Extensive experience with state of art software development practices such as pair programming, Test Driven Development (TDD), refactoring, continuous integration.

Business Analysis and Leadership

• Able to attract, retain and lead people. Led team up to 25 developers, responsible for task planning, technical direction, risk management, code inspections, quality control, performance reviews, staff development. Understood how to create working environment to keep people happy, motivated, creative and productive.
• Solid presentation skill, proven in communication with customers, understanding problems, capturing domain knowledge and turning it into high level requirements in suitable form for development team and other stakeholders

Context root of single war's web applicaton in JBoss

Context root of web application is usually specifiedin META-INF/application.xml in ear file (or a vendor specific jboss-web.xml inside war ) e.g.

...
  <module>
    <web>
      <web-uri>app.war</web-uri>
      <context-root>/jsfejb3<context-root>
    </web>
  </module>
..

However if we drop a single war file without vendor specific descriptor (jboss-web.xml) into JBoss deploy directory then context root is implicitly the name of the war file without extension.
e.g.
Web application jmx-console residing in default/deploy/jmx-console.war has context root jmx-console
Web application groovy-scripts residing in default/deploy/groovy-script.war has context root groovy-script and can be accessible via http://localhost:7080/groovy-script

Centralized Management Tools

Nagios
Big Brother
Hyperic HQ
JOPR
@ctiveFriend
OEM
OpenNMS
There is a great presentation of monitoring tools here

Testing JOPR 2.2.0 - Basic configuration

Modify JBoss server startup options to include JVM as child resource of JBoss Server in JOPR

#$JBOSS_HOME/bin/run.conf
JAVA_OPTS="$JAVA_OPTS -Djboss.platform.mbeanserver"

Remove an uninterested resource from management a.k.a uninventory
Browse Resources->Servers, select uninterested one, hit uninventory button

Testing JOPR 2.2.0 - Installation

What is JOPR, RHQ, JBoss ON
JOPR is an open source project from Redhat to develop complex management platform. JOPR is agent based JAVA/J2EE system, where agent is standalone JAVA running and server runs on top of JBOSS Application Server(AS). It can be used to manage JBOSS AS, Tomcat, Apache as well as Oracle, and any IT resources.

JBoss Operations Network (JBoss ON) is commercial offer of JOPR in form of support service by Redhat.

JOPR is developed on top of RHQ project. RHQ serves as an maven upstream project for JOPR. RHQ consists of GUI management console, agent container and a set of agent plugins for management of common IT resources.
JOPR extends RHQ by adding additional plugins for JBoss App Server, Tomcat Web Server, etc. . JOPR version 2.2.0 is corresponding with RHQ version 1.2.0.

The original of RHQ is Hyperic HQ. Redhat and Hyperic work together to extract common reusable parts of Hyperic HQ and maintain it as RHQ. The aim is to create a common infrastructure for developing management's application. That is why the project named RHQ, where R stand for Redhat.

Install JOPR Server
Download JOPR binary from JOPR website
Follow instruction in Installation Guide to prepare database server for JOPR server
Change working directory to jopr-server-2.2.0 and type

bin/rhq-server.sh start

then open web browser and point to http://localhost:7080
Follow instructions in screen to set database driver, url, username/password to correct values. After that verify database connection.
Hit button Install JOPR server to complete installation process

Modify JOPR server properties file for agent communication
Modify rhq-server.properties file

# jopr-server-2.2.0/bin/rhq-server.properties
rhq.communications.connector.transport=servlet
rhq.communications.connector.bind-address=<jopr-server-ip-address>
rhq.communications.connector.bind-port=7080

Restart JOPR server

bin/rhq-server.sh stop
bin/rhq-server.sh start

Login to JOPR console and watch its log file
Pointing Web browser to http://localhost:7080/Dashboard.do
Login use default rhqadmin/rhqadmin as initial username/password
JOPR log file is located in jopr-server-2.2.0/logs/rhq-server-log4j.log

Install JOPR Agent
This shall be done in all machines that we want JOPR to manage .
Download agent from http://<jopr-server-hostname>:7080/agentupdate/download
Install JOPR Agent by typing

java -jar rhq-enterprise-agent-1.2.0.jar --install

Modify JOPR Agent configuration file

#rhq-agent/conf/agent-configuration.xml

# ask jopr agent not to ask for configuration during startup
<entry key="rhq.agent.configuration-setup-flag" value="true" />

<entry key="rhq.agent.name" value="<jopr-agent-hostname>"/>

<entry key="rhq.agent.server.bind-address" value="<jopr-server-ip-address>" />

# ask jopr agent to listen to on particular ip
<entry key="rhq.communications.connector.bind-address" value="<jopr-agent-ip-address>" />

Start RHQ Agent and watch its log file
change working directory to rhq-agent and run

bin/rhq-agent.sh -c conf/agent-configuration.xml 

If we reinstall JOPR server then agent's local data and server'data is not in sync. In that case we shall ask agent to clean it's local data when start by using

bin/rhq-agent.sh -l -c conf/agent-configuration.xml 

The most recent agent's log file is located in rhq-agent/logs/agent.log