Dump backtrace of all threads in ruby

I have created a few lines of code that allow me to dump stacktrace/backtrace of all threads of a running ruby process. If ruby support Thread#backtrace then it will print backtrace of all threads otherwise it will print of current running thread.
To use it first create a file ruby_backtrace.rb with the following content

require 'pp'

def backtrace_for_all_threads(signame)
  File.open("/tmp/ruby_backtrace_#{Process.pid}.txt","a") do |f|
      f.puts "--- got signal #{signame}, dump backtrace for all threads at #{Time.now}"
      if Thread.current.respond_to?(:backtrace)
        Thread.list.each do |t|
          f.puts t.inspect
          PP.pp(t.backtrace.delete_if {|frame| frame =~ /^#{File.expand_path(__FILE__)}/},
               f) # remove frames resulting from calling this method
        end
      else
          PP.pp(caller.delete_if {|frame| frame =~ /^#{File.expand_path(__FILE__)}/},
               f) # remove frames resulting from calling this method
      end
  end
end

Signal.trap(29) do
  backtrace_for_all_threads("INFO")
end

Then require this file to your ruby script you want to inspect e.g. t2.rb

require 'thread'
require './ruby_backtrace'

def foo
   bar
end

def bar
   sleep 100
end

thread1 = Thread.new do
   foo
end

thread2 = Thread.new do
   sleep 100
end

thread1.join
thread2.join

Finally run the script, send INFO signal to it and look at file ruby_backtrace_pid.txt, where pid is process id

$ ruby t2.rb &
[2] 4719
$ kill -29 4719
$ kill -29 4719
$ cat /tmp/ruby_backtrace_4719.txt 
--- got signal INFO, dump backtrace for all threads at 2012-04-07 17:33:14 +0200
#
["t2.rb:21:in `call'", "t2.rb:21:in `join'", "t2.rb:21:in `
'"]
#
["t2.rb:9:in `bar'", "t2.rb:5:in `foo'", "t2.rb:13:in `block in 
'"]
#
["t2.rb:17:in `block in 
'"]
--- got signal INFO, dump backtrace for all threads at 2012-04-07 17:33:15 +0200
#
["t2.rb:21:in `call'", "t2.rb:21:in `join'", "t2.rb:21:in `
'"]
#
["t2.rb:9:in `bar'", "t2.rb:5:in `foo'", "t2.rb:13:in `block in 
'"]
#
["t2.rb:17:in `block in 
'"]

Simple Python Syntax

Start interpreter and print out something then exit

C:\python
ActivePython 2.5.2.2 (ActiveState Software Inc.) based on
Python 2.5.2 (r252:60911, Mar 27 2008, 17:57:18) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'hello world'
hello world
>>> print 1+2+3
6
>>> exit()

Run a script in the interpreter
To run a script within Python interpreter use execfile(path), it is useful for people who use Jython, because startup time of JVM is horrible . e.g

>>execfile('sample.py')

Create string from a template
This is one of my most frequently used statement, Python follow style of C printf function

>>> "my name is %s, my age is %d" % ("Goto",30)
'my name is Goto, my age is 30'
>>>

Ruby has the same function

irb(main):004:0> "my name is %s, my age is %d" % ["Goto",30]
=> "my name is Goto, my age is 30"
irb(main):005:0>

but there is nicer way to do it

irb(main):005:0> name,age = "Goto",31
=> ["Goto", 31]
irb(main):006:0> "my name is #{name}, my age is #{age}"
=> "my name is Goto, my age is 31"

Create a substring from a string
Python has nice methods get a substring from string

>>> s='hello world'
>>> s[0]
'h'
>>> s[0:10] #substring from a position(inclusive) until other (non inclusive)
'hello worl'
>>> s[2:] #substring to end of string
'llo world'
>>> s[:2] #substring from start until other (non inclusive)
'he' 
>>> s[-2:] #negative position indicates position from end of the string
'ld'

The equivalent in Ruby would be

$ irb
>> s="hello world"
=> "hello world"
>> s[0]
=> 104
>> s[0..0]
=> "h"
>> s[0..(10-1)] #unlike Python, Ruby includes the end position
=> "hello worl"
>> s[2..-1] # -1 indicate relative position from end
=> "llo world"
>> s[0..(2-1)]
=> "he"
>> s[-2..-1]
=> "ld"

Unlike Ruby, Python still throw out of range exception for single indice operation

>>> s[20]
Traceback (most recent call last):
  File "", line 1, in 
IndexError: string index out of range
>>> s[20:30] # this is OK as it consider as slice operation
''

String is immutable
Unlike Ruby, Python string can not be changed directly

>>> s[0:5]
'hello'
>>> s[0:5]="bye"
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'str' object does not support item assignment
>>> s1 = "bye" + s[5:] # to achieve the same goal we need create string from other string 
>>> print s1
bye world

In Ruby we can do

>> s[0..4]="bye"
=> "bye"
>> print s
bye world=> nil

As string is considered value object, create new string instead of changing an existing express that concept more clearly.
List and tuple
Python has tuple and list for representation of variable size array of items. Tuple is immutable array while list is mutable. People think that list is intended for homogeneous while tuple is for non-homogeneous, but this is convention only is not enforced by the language. However some api accept only tuple as argument e.g. String format %, that sometime lead to a confusion. In contras Ruby has only Array.
example of tuple

>>> y=('a',2)
>>> y[1]=1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> y
('a', 2)

example of list

>>> x=['a',2]
>>> x[1]=1
>>> x
['a', 1]

iterate over list or tuple

>>> items = ('a','b',2)
>>> for e in items:
...   print e
...
a
b
2

Hash aka dictionary
Python has hash structure called dictionary

>>> h = {1:'a',2:'b',3:'c'}
>>> h.__class__
<type 'dict'>
>>> h[3] = 'z'
>>> h
{1: 'a', 2: 'b', 3: 'z'}
>>>

iterate over dictionary

>>> for k,v in h.iteritems():
...   print "%d=>%s" % (k,v)
...
1=>a
2=>b
3=>z

h.items() also works well
Class, Object, method
Class is defined using class keyword, name of a class can start lower case or upper case character, which is different from Ruby. Ruby requires name of class start with upper case character.
However the convention is that, name of built-in class as string, unicode, list, tuple start with lower case character while user-defined class starts with upper case.
example of built-in class

>>> y = ('a',2)
>>> y.__class__
<type 'tuple'>
>>> y.__class__ == tuple
True
>>> z = tuple(y)
>>> z
('a', 2)

example of user defined class

>>> class Foo:
...   def m(self):
...     print self.__class__
...
>>>
>>> Foo().m()
__main__.Foo
>>>

Module
Every Python script store in a file is a module, the name of the file is module name.

$ cat hello.py
def say(whom):
  return "hello %s" % whom

To use method defined in a module, just import the module and call the function preceding by the module name plus '.'

>>> import hello
>>> hello.say('world')
'hello world'

It is also common to mix module methods into current name space, so we can call method without typing module name

>>> from hello import *
>>> say('moon')
'hello moon'
>>>

Parameters
Beside normal position based parameters, Python has two extra forms of passing parameters to a function *params and **params.
The first form is argument list, in which caller pass a list parameters and calling function receives them in form of an array.

 def foo(*numbers):
      return sum(numbers)
 
print foo(23, 42)        # prints: 65

The second form is name based, in which caller pass a list name,value pairs and calling function receives then as a hash map.

def bar(**options):
  if "verbose" in options and options["verbose"]:
     print "verbose is ON"
  else:
     print "verbose is OFF"

bar(verbose=True) # print: "verbose is ON"
bar(verbose=False) # print: "verbose is OFF"
bar(force=True) # print: "verbose is OFF"

Closures
Python closures

Faster jruby startup

One of issue with JRuby and Java in general is long startup time. As the result, a lot of peoples including myself use MRI when developing application and then use JRuby to test and run the application.
Recent change is jruby shell script have improved startup time at significantly by putting JRuby library into java boot classpath (using -Xbootclasspath/a:) instead of normal classpath to bypass java class verification, however it still so slow compare to MRI.
Below is a simple test on machine
1. jruby lib in classpath

huy@huy-desktop:/u01/jruby-1.1.1/bin$ time jruby -e "puts 'hello world'"
hello world

real    0m2.934s
user    0m2.604s
sys     0m0.136s

2. jruby lib in boot classpath

huy@huy-desktop:/u01/jruby-1.1.1/bin$ time jruby -e "puts 'hello world'"
hello world

real    0m1.124s
user    0m0.896s
sys     0m0.072s

3. MRI

huy@huy-desktop:/u01/jruby-1.1.1/bin$ time ruby -e "puts 'hello world'"
hello world

real    0m0.060s
user    0m0.004s
sys     0m0.004s

Ruby Virtual Machine's Implementation

link to The Great Ruby Shootout

Nice test code sample using mocha

link to Mock Object Injection
link to Mocking Jruby
link to Boost your Java Test with Ruby and JtestR

Ruby require idiom

Ruby Kernel#require(filename) read and execute the specified file only once regardless how many time we call it as opposite to Kernel#load(filename) which reload the file each time. With application having many files, each depend on others without this behavior of Kernel#require(filename), a single file will be read executed multi time, which may not desired from functional and performance's perspective. The Kernel#require(filename) however has an issue, after loading it puts path of a file required in global array $" and does not know if we refer to the same file using different path. e.g.

#file : lib/foo.rb
puts "loading #{__FILE__}"

#file : lib/bar.rb
require File.dirname(__FILE__) + '/foo'
puts "loading #{__FILE__}"

#file : app/runme.rb

require File.dirname(__FILE__) +'/../lib/foo'
require File.dirname(__FILE__) +'/../lib/bar'
puts $"

Then run

ruby app/runme.rb

loading ./app/../lib/foo.rb
loading ./app/../lib/foo.rb
loading ./app/../lib/bar.rb
["app/../lib/foo.rb", "./app/../lib/foo.rb", "app/../lib/bar.rb"]

There are basically few well known techniques to deal with this problem
1. Using absolute path
2. Modifying $LOAD_PATH
3. Using defined?

USING ABSOLUTE PATH
In this variant we always call Kernel#require with a absolute path, the File::expand_path will be used to remove '..' symbol representing parent directory e.g

#file : lib/bar.rb
require File.expand_path(File.dirname(__FILE__)) + '/foo'
puts "loading #{__FILE__}"

#file : app/runme.rb
require File.expand_path(File.dirname(__FILE__)+'/../lib')+'/foo'
require File.expand_path(File.dirname(__FILE__)+'/../lib')+'/bar'
puts $"

run

ruby app/runme.rb

loading D:/huy/rubyapp/require_1/lib/foo.rb
loading D:/huy/rubyapp/require_1/lib/bar.rb
["D:/huy/rubyapp/require_1/lib/foo.rb", "D:/huy/rubyapp/require_1/lib/bar.rb"]

This method is described in post ruby require idiom

MODIFYING $LOAD_PATH
The second quite popular technique is to modify $LOAD_PATH directly e.g.

#file : lib/bar.rb

libpath=File.expand_path(File.dirname(__FILE__)+'/lib')
$LOAD_PATH.unshift(libpath) unless $LOAD_PATH.first==libpath

require 'foo'
puts "loading #{__FILE__}"

#file : app/runme.rb

libpath=File.expand_path(File.dirname(__FILE__)+'/lib')
$LOAD_PATH.unshift(libpath) unless $LOAD_PATH.first==libpath

require 'foo'
require 'bar'
puts $"

In the previous mentioned techniques, the File#expand_path method is used to get absolute path of either file or directory, the same file or directory is kept only once in a relevant global variable.

USING defined?
This technique is very old and frequently used by C programmers to guard header file to include multiple e.g.

#file foo.h

#ifdef __FOO_H

#define __FOO_H

...

#endif

In Ruby, I have seen this technique being applied using Kernel#defined? e.g.

#file : lib/foo.rb
unless defined?(FooDefined)
   FooDefined=true
   puts "loading #{__FILE__}"
end

#file : lib/bar.rb
unless defined?(BarDefined)
  BarDefined=true
  require File.dirname(__FILE__) + '/foo'
  puts "loading #{__FILE__}"
end

#file : app/runme.rb

require File.dirname(__FILE__)+'/../lib/foo'
require File.dirname(__FILE__)+'/../lib/bar'
puts $"

UPDATE on 26-12-2007
The new Kernel#require in Ruby 1.9 store full path in $" make this article obsolete.

Fixture path in ActiveRecord Testing

I am using fixture in ActiveRecord Testing. I have many test classes, each using fixtures located in different directories. Individual test work fine, but fails when running together in suite. The problem is fixture_path is class attribute/variable of Test::Unit::TestCase so all subclasses of it share the same value.

require 'rubygems'
require 'active_record'
require 'active_record/fixtures'

class TestFoo < Test::Unit::TestCase
   self.fixture_path='/u01/test/fixtures/foo'
end

TestFoo.fixture_path # "/u01/test/fixtures/foo"

class TestBar < Test::Unit::TestCase
   self.fixture_path='/u01/test/fixtures/bar'
end

TestBar.new().fixture_path # "/u01/test/fixtures/bar"
TestFoo.new().fixture_path # "/u01/test/fixtures/bar"

Because the Test Unit framework, load all test classes, then run them, so TestFoo.new().fixture_path get value of TestBar.new().fixture_path. The solution is simple, just define a method fixture_path in each test class (in upcoming version after ActiveRecord 1.15.3, the problem is fixed by change fixture_path to class instance variable see ticket 6672).

class TestFoo < Test::Unit::TestCase
   def fixture_path
      '/u01/test/fixtures/foo'
   end
end

class TestBar < Test::Unit::TestCase
   def fixture_path
      '/u01/test/fixtures/bar'
   end
end

TestBar.new().fixture_path # "/u01/test/fixtures/bar"
TestFoo.new().fixture_path # "/u01/test/fixtures/foo"

Exception handling

I have seen a lot of code, that handle exception in the following way

begin
  do_some_things()
rescue
  #... print message, closing file/socket etc. 
end

There is a problem with this style. If there is error, we lost information where it happens because it is quite common that do_some_things() again call methods of others classes located in different files and so all.
I think that it should be a rule, that we alway print/save to logfile backtrace of exception in every exception handling code

begin
  do_some_things()
rescue => e
  require 'pp'
  pp e
  pp e.backtrace
  #... print message, closing file/socket etc. 
end

The backtrace information will help us to identify bug during development and to analyze root cause of exception during operation.

String concatenation

One of common thing that any programs do is concatenate two or more String value, e.g.

first_name = 'huy'
sir_name = 'le'
name = first_name + ' ' + sir_name # huy le

But the above code suffers a problem, if one of concatenated variables is nil, then it will not work. Checking nil is tedious and error prone, a simple solution of this problem is putting concatenated variables into an array, removing nil using compact and using join method to perform concatenation.

first_name= 'huy'
middle_name=nil
sir_name = 'le'
name = [first_name, middle_name, sir_name].compact.join(' ') # huy le

Standard_params to improve readability of tests

Link for standard_params to improve readability of tests

Ruby block and command/query separation

The recent post Ruby block style do end versus {} on where to use do ... end versus {...} to enclose Ruby block reminds me command/query separation OOP principle.

Easy verification of API behavior via irb

Learning programming language is not just learning language syntax, programmers including myself spend lot of time learning various API. Irb is wonderful tool for learning Ruby API, when I do program in Ruby, I always have Irb open, and I try to run a method of API including core lib, that I am not sure how it works. Comparing to Java, it is much faster and have better productivity.

Open classes

When I want to change or add new behavior to existing class, static typed language give me only two options create new class that either extend existing class or delegate to existing class (in Java world some peoples do aspect oriented programming AOP, but it is little bit complicated and is external to the language).
Ruby give me an ability to modify existing class directly. A cross concern or aspect then can be added easily by modifying existing class.

ADDING NEW FEATURE TO EXISTING CLASS

class Object
   def blank?
      return true if nil?
      return true if respond_to?(:empty?) && empty?
      return false
   end
end
nil.blank? #=> true
''.blank?  #=>true
[].blank?  #=>true
{}.blank?  #=>true
'hello'.blank? #false
[:a].blank? #false
{:a=>1}.blank? #false

The above code define method blank? in class Object, which is root of all Ruby classes. This will result in well behaving instances of NilClass, String, Array, Hash.

DECORATION A METHOD OF EXISTING CLASS
To decorate a method of existing class is just easy as adding new one e.g.

require 'active_support'
class Hash
    alias original_symbolize_keys! symbolize_keys!

    def symbolize_keys!
        each_value do |value|
             value.symbolize_keys! if value.is_a?(Hash)
        end
        original_symbolize_keys!
    end
end

Rails ActiveSupport add method symbolize_keys!, above code decorate it to convert keys to symbol for a hash value if it is also hash.
When modifying an existing class, we shall make sure that the class is already loaded, otherwise we may get a strange behavior. This is specially important in rails, when classes are loaded dynamically using const_missing method. About how to handle this problem, there is good tip reopen with class/module eval on practical ruby blog. Other way to decorate method without using alias is available on replacing methods.

Using of keyword based parameter

I plan to have presentation of Ruby Language in enterprise application development for developers of the company I worked for and I am thinking about which message should I pass to them. Instead of trying to give many reasons that others already talked about huge benefit of using Ruby, I decide to give my own very personal reasons why I love Ruby based on my one and half year working with this language.
In early day, when I programed in PL/SQL, I know that PL/SQL support keyword based parameter in procedure/function but I never used this feature, simply I have not recognized the benefit of using it that time.
Ruby does not support keyword based parameter but Ruby programmers fake it easily using hash in combination with symbol. Using this style is very popular in Ruby core lib and Rails. The following example illustrates it, suppose we want to create method create_user that create database user, in traditional position based parameter, we will do like that

def create_user(username,password,ignore_error,force,verbose)
   #.. implementation detail is ignored
end
#calling it
create_user('scott','tiger',false,true,true)

on keyword based version, the method has simply one parameter params

def create_user(params)
   username = params[:username]
   password = params[:password]
   ignore_error = params[:ignore_error]
   verbose = params[:verbose]
   force= params[:force]
   #.. implementation detail is ignore
end
#calling it
create_user(:username=>'scott',:password=>'tiger',:ignore_error=>false,
  :verbose=>true, :force=>true)

The keyword based version is obviously more verbose, requires more typing, but offers several benefits.

MORE EXPRESSIVE AND LESS ERROR
The keyword based version is more expressive, just by looking at how the method is called, we know what is the intention, there is no need to look at the implementation file to figure out what it does. The position based version suffers what Joshua Bloch mentioned in his How to Design a Good API and Why it Matters "Long lists of identically typed params harmful"

GOOD DEFAULT
In the position based version, we can only assign default value for those parameters that are at the end of parameter list.

def create_user(username,password=username,ignore_error=false,force=false,verbose=false)
   #.. implementation detail is ignored
end
#calling it
create_user('scott')

This will not give a flexibility of using default value just for few last one. In the keyword based version, we can archive it easily as follow

def create_user(params)
   username = params[:username]
   password = params[:password] || username
   ignore_error = params[:ignore_error] 
   verbose = params[:verbose]
   force= params[:force]
   
   #implementation detail is ignored
   
   #note that Ruby consider nil as false in condition expression, so we should design 
   #boolean value keyword in such way that its default value is false
end

#calling it
create_user(:username=>'scott',:verbose=>true)
create_user(:username=>'scott',:force=>true)
create_user(:username=>'scott',:ignore_error=>true)

I also see people using Hash::merge to shorten assignment of default values and adding some assertion of accepted keywords e.g

#borrow from ActiveSupport
class Hash
  def assert_valid_keys(*valid_keys)
    unknown_keys = keys - [valid_keys].flatten
    raise(ArgumentError,
    "Unknown key(s): #{unknown_keys.join(", ")}\nValid key(s): #{valid_keys.join(',')}")   
      unless unknown_keys.empty?
  end    
end

class DatabaseSchemaBuilder
  
  attr_accessor :ignore_error,:verbose,:force

  def default_options
    {:ignore_error=>@ignore_error,:verbose=>@verbose,:force=>@force}
  end

  def create_user(params)
    params.assert_valid_keys(:ignore_error,:verbose,:force)
    params = default_options.merge(params)
   
    username = params[:username]
    password = params[:password] 
    ignore_error = params[:ignore_error]
    verbose = params[:verbose]
    force= params[:force]

   #implementation detail is ignored
  end
end

EASY TO CHANGE
When we need lets say adding new parameter to the method. In position based version, we end up with changing contract of the method, which may result in looking at every line of code that use this method and make change unless you put at the end of parameter list with default value.
In keyword based version, it is obvious less painful, just adding one more keyword, setting a default value, anyway change only the method itself, e.g. we want to add parameter :noop, meaning no operation, just for testing.

def default_options
   {:ignore_error=>false,:verbose=>false,:force=>false,:noop=>false}
end

def create_user(params)
   params.assert_valid_keys(:ignore_error,:verbose,:force,:noop)
   params = default_options.merge(params)
   
   username = params[:username]
   password = params[:password] || username
   ignore_error = params[:ignore_error]
   verbose = params[:verbose]
   force= params[:force]
   noop=params[:noop]

   #implementation detail is ignored
end

HIDING DESIGN DECISION
Using position based parameter with little bit long parameter list, I have to decide if put one parameter before other or not. I do not have this problem when using keyword based parameter, so it help me do program faster.

Ant depend task

I am writing ruby helper class that will be used in Rakefile to build complex java application. Everything seem to be quite simple, I just copied what has been done by Matt Foemmel in his JRake. The ruby helper class has a method that check if java class is upto date by comparing access time of the class file and the java source file. I believe that Ant javac task does the same.
Using the ruby helper file, my java application most of time get build correctly but sometime, few class files are not upto date. It take me a while to figure out what is going wrong.
The problem is when a java class depends on other, if other class changes then the java file shall be recompiled even though it is not modified. Looking at Ant documentation, I found that Ant solve this problem using depend task.
In order to solve this problem in ruby, I have to do the same as Ant depend task does, read our class file, extract all class references verify if these class references are changed.

irb and TAB autocompletion

To enable autocompletion on irb, run

irb
require 'irb/completion'

Note to hit double TAB on your keyboard not single one. One of most comprehensive description can be found here

gem's environment

When playing with gem on my Ubuntu machine, I found an useful command

huy@huy-desktop:~/.gem$ gem environment 
Rubygems Environment:
  - VERSION: 0.9.0 (0.9.0)
  - INSTALLATION DIRECTORY: /var/lib/gems/1.8
  - GEM PATH:
     - /var/lib/gems/1.8
  - REMOTE SOURCES:
     - http://gems.rubyforge.org

that shows default gem's environment. By setting the GEM_PATH environment variable to one desire directory, we can share single gem repository across many ruby installations (e.g C ruby and JRuby). The full descriptive information of all gem environments variables can be found in Gem Command References

Cruisecontrol.rb is too slow

I encountered a problem with Cruisecontrol.rb, the method link_to_code(log) is too slow to process a certain output log, sometime it takes 10 seconds making Cruisecontrol not workable. So for the time being, as workaround, I just comment out this code .

#file: app/helpers/builds_helper.rb
  def link_to_code(log)
    log
=begin    
    @work_path ||= File.expand_path(@project.path + '/work')

    log.gsub(/((\#\{RAILS_ROOT\}\/)?([\w\.-]*\/[ \w\/\.-]+)\:(\d+))/) do
      path, line = File.expand_path($3, @work_path), $4
      
      if path.index(@work_path) == 0
        path = path[@work_path.size..-1]
        link_to ".#{path}:#{line}", "/projects/code/#{@project.name}#{path}?line=#{line}##{line}"
      else
        $1
      end
    end
=end
  end

Ruby adoption in Vietnam

From the time, I know ruby, it is my favorite language, but it seems that no so many developer in Vietnam is using it. Today I have found Vietnam Ruby site. It is good news, that ruby got an attention in Vietnam and I am not alone, who see the huge benefit of wonderful and humane programming language.

ruby module_function

Ruby 'module' allows us to define methods then turn these methods into instance methods of a class by using key word 'include'. In ruby world, peoples call this feature 'mixin'. e.g.

module Foo
   def hello
      puts 'hello'
   end
end
class Bar
include Foo
end
Bar.new.hello #=> hello

Sometime we need to call method defined in a module without including the module into a class. We can do it by

module Foo
  def self.world
     puts 'world'
  end
end
Foo.world #=> world

There is a question how do we turn method 'hello' so it can be used in both mixin style and standalone . The answer is using kernel method 'module_function'

module Foo
   module_function :hello
end
Foo.hello + ' ' + Foo.world # => hello world