Migrate configuration to Chef

One of big headache in implementation Chef or any automate configuration management tool is migration. We usually have to face with a large legacy configuration that is poorly designed and implemented, which make thing worse.
I want to share some of our own experiences in doing migration of a relatively large configuration in term of both scale and complexity.
The tools and techniques
They are
1. a decent, fast and rock solid version control system: I recommend git.
2. semantic diff tools: diff of text file is OK, but not enough, depending on format of your configuration (XML, properties, yaml, JSON), you may need to write your own semantic diff, that is able to make intelligent diff of these formats.
3. source code of these applications that use these configuration: the migration involve legacy applications, access to source code is absolute need to order to do a refactoring.
4. collaboration: with the new tool developers need to change/cleanup the code that access configuration data, operation people need to change their practices, we all need to learn how to use, what to trust, where we need to pay attention.
Shadow configuration file
In order to minimize impact existing system, at the beginning we let chef to create shadow configurations, which then are verified with existing configuration using diff tools. The shadow configuration generated by chef will replace the actual one only after we make sure that they are semantically the same from the point of view of the program using it.
In shadow configuration we can have a configuration file with different prefix/postfix or different directory. e.g. if the actual configuration file is /etc/hosts then the shadow one will be /etc/hosts.chef or /etc-chef/hosts or /root-chef/etc/hosts. I personally prefer the complete shadow directory because it is easier to check, remove when needed.
The cookbook need to have one attribute (can be name of directory where we are going to generate configuration files) saying if we are going to generate shadow or actual configuration.
Conflicting changes
In ideal world, we just has one configuration management system and one administrator who modify the system. But reality is different. There are usually more actors that make the change. So there is potential conflict. E.g. Chef modify one file and then an administrator or scripts or other configuration system don't know and also modify it. The worst case is that everyone assume that the file has the content he expect and thing get broken. These scripts that work for many years suddenly fails.
How to minimize the conflict?. Communicate well with other team members what are managed by Chef, which they should modify through Chef and not manually or by other tools. Various methods can be used e.g. a) wiki pages, b) put a clear notice as comment at the beginning of all files managed by Chef such as "This file is generated by Chef, you shall not modify it as it will be overwritten" c) notify all people each time a file is modified by Chef.
Top-down vs. bottom-up
There is basically two approaches to start the migration project 1) top-down and 2) bottom-up.
1. In top-down approach, you start with create a cookbook/recipe for one configuration item (e.g. syslog-ng, the cookbook will be fairly generic so it is usable by all environments) . Then extract specific data (e.g. ip of syslog server, source of log) from configuration file from each server and put them as attributes into a role. After finishing one configuration item, you continue with other until you have all you need in Chef.
2. With bottom-up approach, we first create one generic cookbook/recipe per server role (e.g. middleware) then you add all configuration files from servers of all environments to it. The structure of cookbook files can be like

      middleware/files/default   # files that are same for all hosts of all env.
      middleware/files/default/dev   #  files that are same for all hosts of development env.
      middleware/files/default/dev/host1   # files that are specific for host1 of evelopment env. 
      middleware/files/default/dev/host2
      middleware/files/default/qa
      middleware/files/default/pro

The recipe can be easily developed to copy these configuration files to a server depending on environment and hostname (node[:hostname]). After everything are in Chef and work well, we will start refactor and split this big generic cookbook to many smaller cookbooks, adding more roles and make thing easily to reuse. This step is kind of continual improvement process aiming at making our chef repository better and better.

In our project we have taken the first approach because it seems more naturally at first sight, however later we change to the second one, which I think have many benefits.
With the second approach, after we put all things in the Chef (can be done pretty quickly), we can announce that from now all changes has to be done through Chef. The very first result is seen immediately: a) there is single point of making change b) change is well audited and communicated as Chef repository is under version control system c)conflicting changes is minimized

Chef - Different between method defined in recipes and libraries

There is one thing that need an attention when developing a recipe. If we create a method in recipe then that method is not available inside resource block's parameter. I will show it in the following example using Shef, the Chef interactive Ruby console.

[root@localhost gems]# shef

Ohai2u root@localhost!
chef > recipe
chef:recipe >   # here we are inside of a recipe
chef:recipe > def account(path) # create a method
chef:recipe ?>    return 'nobody' if File.dirname(path) == '/tmp'
chef:recipe ?>    return 'root'
chef:recipe ?> end
 => nil 
chef:recipe > file "/tmp/file.test" do
chef:recipe >      action :create
chef:recipe ?>     owner account(name) # try to use this method inside the block and we got a error
chef:recipe ?> end
NoMethodError: undefined method `account' for Chef::Resource::File
 from /usr/lib/ruby/gems/1.8/gems/chef-0.9.8/lib/chef/resource.rb:84:in `method_missing'

One way to workaround is to call the method outside of resource block's parameter so the method is called in the context of recipe

chef:recipe > fname = "/tmp/file.test"
 => "/tmp/file.test" 
chef:recipe > account_val = account(fname)
 => "root" 
chef:recipe > file fname do
chef:recipe >   action :create
chef:recipe ?>  owner account_val
chef:recipe ?> end
chef:recipe >

Other way is to define it as library (put it inside the directory libraries), in shef we would do like that

chef:recipe > exit 
 => :recipe
chef >  # here we are in top context, in which libraries are loaded 
chef > def account(dirname)
chef ?>    return 'nobody' if File.dirname(path) == '/tmp'
chef ?>    return 'root'
chef ?> end
 => nil 
chef > recipe
chef:recipe >   # here we are inside of a recipe
chef:recipe > file "/tmp/file.test" do
chef:recipe >    action :create
chef:recipe ?>   owner account(name) # now it is OK
chef:recipe ?> end
chef:recipe >

In that case, the method is visible in the context of resource.

Some experiences in using Opscode Chef

I am reaching the end of our Chef's implementation project, in which Chef is deployed to manage hundreds of Apache and Jboss servers (Redhat) hosting complex and dynamic banking application.
So it is worth-wise to recap some issues, problems encountered and solutions employed during the course of implementation.
Chef vs other tools
The project actually started from around Aug/Sep last year with the decision to start with Chef instead of other tools Puppet, CF Engine. I choose Chef because CF Engine is just too old while Puppet is too complex, both require to learn new language for expressing configuration and they missed GUI. Other quite subjective reason is that I know Ruby better comparing to other.
The challenges
The technical challenges include the complexity of the configuration being migrated to Chef as well its ever changing nature during the course of implementation. As Chef represents paradigm's shift to what we called infrastructure as code, the most difficult part is to persuade the team to change their habit and to adopt software development practices in their work.
Refactoring
Our environment is complicated and legacy. It was created over years by different guys manually with a bundle of shell scripts, that are far from good in quality from software engineering perspective (consistency, less duplication, good naming). So the implementation also mean refactoring the environment to something more viable in order to automate.
Cookbooks, roles are developed in parallel with renaming of configuration files/directories, cleaning up obsolete stuffs while adding support for new developed applications, environments, at the same time making sure not to break any things.
Single vs multi instances
At the beginning of the project there is no environment supported in Chef so there is two options a) maintain separate Chef instance per environment b) embed environment's information into role. As we have too many environments (development, preproduction, quality assurance, training, laboratory, DRS, production), it would be less work to select second option.
Cookbooks are developed and shared between all environment while roles are created per environment. I use e.g. the following naming convention for role: dev_apache_internet_os to denote role containing attributes for OS related cookbooks of Apache server in Internet zone.
Roles as Ruby code
We have nearly hundred of roles, a server is assigned to at least 2 roles one of OS and other of application specific. That is for separate the concern. OS role will has attributes more less related to OS e.g. pam, ssh, hosts, dns, ntp, postfix, route while application specific are e.g. Apache, Java, Tomcat, Jboss, log4j, etc.
To make it easy to manage; a subdirectory is created for each environment and roles are placed in it. The role format is Plain Old Ruby instead of JSON. This is because comparing to JSON format, the Ruby one is easier to spot error saving a lot of time. When we have hundred lines in a Role's file, it is quite possible that we make some typing errors.
The other reason is that we can do some sort of programming in Role file e.g. it is more convenient using a loop to create mod_jk config that route request to dozen of servers.
One layer above roles
As the number if roles grows, I have seen that many roles of different environments share common attributes. At OS level e.g. we use same DNS and NTP for both production and non production just in case of DRS there is different. Such examples are endless. These attributes are mostly not same in all environments, they are just same in some so it is not wise to put them in attributes's file of relevant cookbooks. Therefore we have created additional layer to support reduce such kind of duplication. These attributes and their values are kept together with applicable environments e.g. in separate file

set_config_item(:item=>"dns",:env=>["dev","pre","qa"],:value=>["192.168.0.1","192.168.0.2"])

and in the Role' file, we fetch the relevant attributes value. e.g.

"dns"=>get_config_item(:item=>"dns",:env=>"dev")

Call Attribute#to_hash in recipe
Cookbooks are commonly used in all environments. Beside infrastructure, we also create cookbooks that maintain our application specific setting such as URL of external partner systems that we communicate with. The fundamental structure are instances and application modules. Many different application modules can be deployed in single instance and the same application module can be deployed in different instances to support of different versions of the same module and dedicate instance per customer delivery channel.
Recipe of a cookbook get data from a role to define resources (file, directory, permission, etc.). As we want make role simple, easy to create and modify, I put more logic in recipe. Some recipes are real programming stuff that need to iterate over structure of a role to get desire data. The autovivify feature of attribute sometime causes problem, in order to avoid it we have to call to_hash for attribute before doing any iteration over it.
Avoid putting big binary in a cookbook
We initially tried to put all files required by a recipe in its cookbook. It turn out to be a bad idea to keep e.g. 200 MB of Jboss installation in Couchdb of Chef. Both knife and Chef-client runs very slow and un reliably. At the end we decided to keep these big binary (more than 20 MB) outside Chef cookbook and setup an Apache for handling these files. The recipe then has to be written to handle checksum, downloading (we use wget) and installation.
We do not have to do it in case of software packages available as rpm, because chef package resource supports their installation.
Make sure that there is no non-ascii character in /etc/passwd,group
chef-client use ohai to parse certain OS configuration files e.g. /etc/password to JSON, if these files contain non-ascii characters, it can causes a problem because JSON lib can not cope with that. So it is better to remove non-ascii from OS configuration files when migrating existing machine to chef.
Disable SELinux
SELinux se store security context per files & directories. When chef-client modify configuration files , it may not retain that information making some daemon not working properly. Chef does not support SELinux so we need to disable it to avoid problem.
Start multi chef-server-api instances for better performance
A default chef-server can not support hundreds of clients, so we installed front end Apache that handles SSL encryption for chef-client as well as balances request to 4 instances of chef-server-api, which we start using option -c 4.
run chef-client as root in crontab
We run chef-client on our non production environments, this is because we want to avoid possible problem with long running ruby process and to reduce the chance that chef-client at many machines run at one moment causing high load to chef-server.
Using crontab we can easily specify time when they will run for each group of machines or even machine. Time to run chef-client can be further generated automatically using hash of machine name to achieve a fair distribution of load.
For user, group management chef-client uses unix command lines, so chef-client must run as root with appropriate path to locations of useradd, groupadd, … (e.g. /usr/sbin,/usr/bin,/bin, /sbin).
The flow of change
The chef-client does not run automatically in production one. This is because in production we want more control as well as due to our change management policy. It is also safer taking into consideration that modification of cookbook can potentially impact production.
The flow of change in production (after cookbook being modified and tested in other environments) involes 1) modify relevant production role, 2) take one production machine from service, 3) execute chef-client on that machine, 4) verify that it work as per expectation, 5) execute chef-client in all remain servers and put all into service.
To execute the chef-client command on a group of machines, we can use sort of Command Control system (e.g. rundesk, control tier, ..) or simple knife.
Hacking, Notification and improvement
To increase safety in the context of missing noop option in chef-client, I have create a monkey patch for certain providers so when chef-client modify a file it notify us by sending output of diff between new and old file by e-mail. The custom notification also send alert to certain group of people depending of nature of change and affected environment.
Beside that, we have create new resources and change behavior of fews existing one to suit more our need. The code base of our cookbook, roles is now reaching 20K line of Ruby code and we continually adding more thing as well improve it.

chroot - Change Root

I have recently closed a year old ticket that has some thing to do with chroot. For security reason, we use chroot (in modsecurity) to restrict our Apache process to access only to a desired directory tree. As per chroot document, a program that is re-rooted to another directory cannot access or name files outside that directory, and the directory is called a "chroot jail".
We make sure that chroot is called after Apache process complete it' initialization in order to not break anything, Because otherwise Apache will not be able to access needed share lib, log files, pid file located in various system directories.
However there is a wrong access time the log produced by our Apache. It is always GMT not local time as we want. We have opened ticket with vendor, searching over internet, looking at source code but could not figure out why. The worse thing is that, for first few requests , the access time is correct (local time) but then it gets change to GMT.
Yesterday I have found the reason. I remember that when I ran the strace with the Apache process being chrooted, I saw Apache try to open some files but could not find it. We see a lot of file not found when running strace because various libraries intend to open some file and if it is not found then try others. But in that case the file is /etc/localtime. So it turns out that when logging Apache use apr lib, which call gmtime and mktime, which need to access /etc/localtime. So missing this file in "chroot jail" causes a problem with access time. Without file /etc/localtime gmtime and mktime consider that the machine is in GMT time zone.

Using debuginfo packages in Redhat

Redhat provide debuginfo packages but not tell us so much how to use it. Here is a simple instruction to use debuginfo packages.
First this is important to know that a debuginfo package contains only debug symbol not the executable. Therefore we need to install both normal package and the corresponding debuginfo and of course their version must be matched.
Installation
Make sure that repository of debuginfo package is configured properly as follows

[root@localhost ~]# cat /etc/yum.repos.d/rhel-debuginfo.repo 
[rhel-debuginfo]
name=Red Hat Enterprise Linux $releasever - $basearch - Debug
baseurl=ftp://ftp.redhat.com/pub/redhat/linux/enterprise/$releasever/en/os/$basearch/Debuginfo/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

To install debuginfo package, we have to specify --enablerepo rhel-debuginfo as yum option

root@localhost ~]#yum install --enablerepo rhel-debuginfo httpd-debuginfo

root@localhost ~]#yum install httpd

[root@localhost ~]# rpm -q -a | grep httpd
httpd-debuginfo-2.2.3-43.el5_5.3
httpd-2.2.3-43.el5_5.3

Usage
To get use just start gdb session with a normal executable

[root@localhost ~]# gdb /usr/bin/ab
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/bin/ab...Reading symbols from /usr/lib/debug/usr/bin/ab.debug...done.
done.
(gdb) break main
Breakpoint 1 at 0x7402: file /usr/src/debug/httpd-2.2.3/support/ab.c, line 1900.
(gdb) run
Starting program: /usr/bin/ab 
[Thread debugging using libthread_db enabled]

Breakpoint 1, main (argc=114424, argv=0x0) at /usr/src/debug/httpd-2.2.3/support/ab.c:1900
1900 {
(gdb) l
1895 
1896 /* ------------------------------------------------------- */
1897 
1898 /* sort out command-line args and call test */
1899 int main(int argc, const char * const argv[])
1900 {
1901     int r, l;
1902     char tmp[1024];
1903     apr_status_t status;
1904     apr_getopt_t *opt;
(gdb) 

M Language Tutorial - The cryptic style

In order to save typing at cost of readability, every command and built in function of M has abbreviation/shortcut.
The following examples are perfectly valid

f i=1:1:9 s sqtable(i)=i*i; 'f' stand for 'for' and 's' for 'set'

and

s i="" f  s i=$o(sqltable(i)) q:i=""  w i," ",sqltable(i),!;'q' stand for 'quit' '$o' for '$order', 'w' for 'write'

As we spend most of our time reading code written by others, modern programming style does care much about naming thing. The abbreviated style of most if M code is really pain especially for novices.

M Language Tutorial - for and Array

for command

GTM>for i=1:1:5 write i,!
1
2
3
4
5

The above for loop a variable i from 1 to 5, increasing by 1 in each iteration.
This one will loop over a list of arguments

GTM>for i="hello","world","bye","moon" write i,!
hello
world
bye
moon

array
In M array is stored as sparse B-tree structure, index or subscript can be anything and can be in any number

GTM> 
GTM>set a(1)="hello",a(2)="world" 
GTM>write a(1)," ",a(2)
hello world
GTM>
GTM>write a(3)         
%GTM-E-UNDEF, Undefined local variable: a(3)

the index of an Array can be any thing so it is like hash/dictionary in other language.

GTM>set a("hello")=1

GTM>set a("world")=2

GTM>write a("hello")
1
GTM>write a("world")
2
GTM>write a("moon") 
%GTM-E-UNDEF, Undefined local variable: a(moon)

We can use any number of subscripts

GTM>set a(1,2)="moon"

GTM>set a(1,1)="world"

GTM>write a(1,1)
world
GTM>write a(1,2)
moon
GTM>write a(1,0)
%GTM-E-UNDEF, Undefined local variable: a(1,0)

built-in function $order is used to get index/subscript of an element of an Array, that is particularly useful for traversal over an Array.

GTM>set b(3)="hello",b(5)="world",b("hello")=1,b("world")=2

GTM>write $order(b("")); with empty string we get a subscript of the first element of an Array
3
GTM>write $order(b(3)); passing one element we get a subscript of next element  
5
GTM>write $order(b(5))
hello
GTM>write $order(b("hello"))
world
GTM>write $order(b("world")); passing the last element we get empty string 

If array is multi dimensional, $order will return subscript of an element in one dimension

GTM>set a("h",1)="hello",a("w",1)="world"

GTM>write $order(a("")); return subscript in the first dimension of first element 
h
GTM>write $order(a("h")); return subscript in the first dimension of second element 
w
GTM>write $order(a("w"))

GTM>write $order(a("h","")); return subscript in the second dimension of first element  
1
GTM>write $order(a("h",1)) 

GTM>write $order(a("w","")); return subscript in the second dimension of second element 
1
GTM>write $order(a("w",1)) 

traversal over a array
for is typical command used to traversal an array

GTM>kill a for i=1:1:10 set a(i)=i*i

We use kill to erase content of the variable a if exists and a for to create 10 elements.

GTM>set i="" for  set i=$order(a(i)) quit:i=""  write i," ",a(i),!; two SPACE after for and two SPACE after quit:i=""
1 1
2 4
3 9
4 16
5 25
6 36
7 49
8 64
9 81
10 100

Here, the argument-less For repeats until stopped by a terminating quit. This line prints a table of i and a(i)
block of multi line of codes under for
M support execute multi line of code under for in the routine

GTM>zedit "nested"
finish
 for name="ivan","john","janes" do
 . if name]"jo"; the operator ] test if name follows string "jo"
 . else  write name,!
 quit

GTM>zlink "nested"

GTM>do finish^nested
ivan
janes
GTM>

M Language Tutorial - if, else and boolean condition

M language has no concept of structure that are familiar in other programming languages. In M all are commands including if,else,for.
if and else commands

GTM>set x=1

GTM>if x=1 write "boom"; if condition is satisfied
boom
GTM>if x=2 write "bang";  if condition is not satisfied

The if command support AND , OR, NOT condition

GTM>set x=1,y=2

GTM>if x=1,y=1 write "bang" ; we use , to denote AND

GTM>if x=1,y=2 write "bang"
bang

GTM> if (x=1)&(y=2) write "bang"; other way to express AND is to use &
bang

GTM>if x=1!y=1 write "boom"; we use ! to denote OR 
boom
GTM>if 1=0 write "oh" 

GTM>if '1=0 write "oh"; we use '  to denote a negation
oh

Internally, if set a special value $TEST to 1 (TRUE) or 0 (FALSE) depending on whether the condition is satisfied or not. $TEST is set to 1 at the start of gtm. The if command without condition will use $TEST as condition.

$gtm
GTM>write $TEST
1
GTM>if 1=0

GTM>write $TEST
0
GTM>if  write "hello"! note two SPACEs after if

GTM>if $TEST=1 write "hello"! equivalent to above

GTM>if 1=1

GTM>write $TEST
1
GTM>if  write "hello"; note two SPACEs after if
hello
GTM>if $TEST=1 write "hello"; equivalent to above
hello

else is command that perform the command followed it if the $TEST variable is 0

GTM>set x=1

GTM>if x>1 write "world"

GTM>else  write "moon"; note two SPACES after else
moon
GTM>if $TEST=0  write "moon"; equivalent to above
moon

command's postcondition
execution of almost all commands can be controlled by following it with a colon and a truthvalue expression.

GTM>set n=1
GTM>write:n>0 "hello"
hello
GTM>write:n>1 "world"
GTM> 

block of multi line of codes under if,else
M support execute multi line of code under if,else, for in the routine

GTM>zedit "nested"
start(x,who)
 if x=1 do  ;two SPACE after do
 . write "hello",!
 . write who
 else  do  ;two SPACE after else and do
 . write "bye",!
 . write who
 quit

GTM>zlink "nested"

GTM>do start^nested(1,"world")
hello
world
GTM>do start^nested(0,"world")
bye
world

operators for string' comparison
These operators are a little bit strange comparing to other languages

GTM>WRITE "A"="B"; compare if two string are equal
0
GTM>WRITE "C"="C"
1
GTM>WRITE "A"["B"; this is same as contains in other language
0
GTM>WRITE "ABC"["C"
1
GTM>WRITE "A"]"B"; this is same as > in other language
0
GTM>WRITE "B"]"A"
1

NOT operator can applied to either expression or other operator

GTM>write "A"="B" 
0
GTM>write '("A"="B")
1
GTM>write "A"'="B"
1

M Language Tutorial - routines

Divide and conquer is the technique used to counter complexity from the beginning of software development history, A complex system in M language comprise of many files called routines.
label, quit and function
In general each line of code in a routine contains of one or no label following by a SPACE or TAB and then M command, other M code can make a call to any line of the routine that has a label at start of the file. Let create a file display.m in one of these paths specified in env. variable gtmroutines using any editor.

start
 write "display",!
 quit
other 
 write "other",!
nothing
 write "no",!

Now start gtm and try to make some call

$gtm
GTM>zlink "display"
GTM>do ^display
display
GTM>do start^display
display
GTM>do start+1^display
display

The name after ^ is name of routine same as filename without extension .m. Without any label, GTM will execute the code starting from line 1 of the routine. It does exactly the same with label start. Making a call to a line 1 from the label start mean start the execution from line 2 of the file. The execution terminate at command quit.

GTM>do start+3^display
other
no
GTM>do other^display  
other
no
GTM>do nothing^display
no

Calling a line 3 from label start is the same as calling label other. Because there is no quit command, GTM continues the execution at the end of file.
There is good practice always structure a routines as series of sections starting with a label and ending with a quit command. That way we can consider each label as a function name when making a call. Using offset from a label is considered a bad practice as it decrease readability of the code.
When calling a function within the same routine, we can remove the ^filename

talkto(who)
 do say("Hello",who); call a function in the same routine
 write "bla bla",!
 do say("Byte",who)
 quit

say(what,who)
 write what," ",who,!
 quit
GTM>zlink "stuff"

GTM>do talkto^stuff("John")
Hello John
bla bla
Byte John

calling function, passing parameters, return value
We pass parameters when calling M routine in parentheses separated by comma, the leading period . is used to indicate a parameter being passed as reference that is used to store the output of function.

$gtm
GTM>zedit "calc"
calc(a,b,ret)
 set ret=(2*a)+(3*b)
 quit

Now make a call

 
GTM>zlink "calc"
GTM>do ^calc(1,2,.result)
GTM>write result
8

M provides a facility called Extrinsic Variable to create a function that return value so it can be used in a expression

$gtm
GTM>zedit "calc"
calc(a,b,ret)
 set ret=(2*a)+(3*b)
 quit

othercalc(a,b)
 quit (2*a)+(3*b); the function must put return value  after quit command

GTM>zlink "calc"
GTM>write $$othercalc^calc(1,2); put $$ before the name of function when calling
8
GTM>set x=$$calc^calc(2,3)     
%GTM-E-QUITARGREQD, Quit from an extrinsic must have an argument

M Language Tutorial - Getting started

I am learning MUMPS/M language to understand one of our systems, that has been implemented using M. I feel, there is a lack of a documentation in form of quick start and tutorial, so I put some notes here hoping that it will be useful for someone.
For a thorough listing of the M language commands, operators, functions and special variables, see MUMPS by Example
One of M implementations available as open source is GTM from FIS, we can download it from http://sourceforge.net/projects/fis-gtm/. The documentation is available from here
Setup a environment to start some test is fairly simple, just unzip the downloaded file, run configure and answer few questions, then we can start
setting env. for running GTM

$source ./gtmprofile

This shell will set various env, variables required by GTM also create a global directory and a default datafile, that is actually M database. M is a language with built in persistence. The global directory is kind of control file in Oracle.
start gtm - the interpreter

$gtm
GTM>
GTM>write "hello world"
hello world
GTM>halt

M use write command to output some thing to a console, in Ruby we would use puts in Python, this is print. The command halt is used to quit the gtm.
For a beginner of any language, the ability to write something out to see and to exit are the two most important commands.
string and numbers
String is enclosed by " as in C, Java, Python, Ruby, to concatenate two or more string we use operator _, not that common any more.

GTM>set x="hello"
GTM>set y="world"
GTM>set z=x_" "_y
GTM>write z
hello world

The command set set these local variables, these variables are created automatically if not exist.

GTM>set a=10
GTM>set b=20
GTM>set c=(a+b)*5/10
GTM>write c
15

For number, this is quite straightforward, no surprise.

create and run the first program

$export gtmroutines=/home/gtm/samples
$source ./gtmprofile
$gtm
GTM>write $ZROutines
/home/gtm/samples

this gtmroutines env. variable specifies where GTM is going to store and search for its routines, inside GTM however it is kept in variable $ZROutines. The concept routine in GTM is synonym for a single file containing M code.

GTM>zedit "hello"

This will popup a vi editor with opened file /home/gtm/samples/hello.m. Create the following content, save and quit vi editor.

hello(who)
  write "Hello ",who,!
  quit

In the routine hello.m we see a label hello(who), which is calling entry point. GTM's routine can have many labels and the general syntax of calling a section of code in the routine is label^routine. The label hello will take one parameters. The write above command takes 3 parameters separated by comma where ! means new line. It is possible to write the above code in a single line

hello(who) write "Hello ",who,! quit

The quit at the end is important in case other labels are added below if we want a label to behave as a function, which mean that the execution flow in the routine terminates at quit command .

GTM> zlink "hello"

The zlink compiles and link hello.m to GTM image so we can call this routine in GTM environment

GTM>do hello^hello("Moon")
Hello Moon

If we do not specify a label then the code will be executed from the first line.

GTM>do ^hello("World")
Hello World

If we look closely to gtm, we will see that gtm is shell script that call the binary mumps in direct mode (with parameter -direct).
We can run the routine hello.m from command line as follow

$export gtmroutines="/home/gtm/samples/ ."
$mumps -run %XCMD 'do ^hello("World")'

The %XCMD is in fact a routine _XCMD.m located in distribution directory of GTM, that is why we need add "." into gtmroutines env. variables, so mumps knows where to find the files being executed.
language syntax
M syntax is very strict, SPACE characters between M statements are significant. A single space separates a command from its argument, COMMA "," is used to separate one argument from other of these commands taking more than one arguments.
A SPACE, or newline, separates the command's arguments from others. Commands which take no arguments (e.g., ELSE) require two following spaces.
Character ; is used to indicate start of a comment, that runs until end of line.

GTM>write  "hello"; two SPACES after write
%GTM-E-EXPR, Expression expected but not found
 write  "hello"
       ^-----
GTM>write "hello"; single SPACE after write
hello
GTM>set x=1,y=2 write "x=",x,",y=",y !set and write commands are on the same line separated by SPACE
x=1,y=2
GTM>set x=1 + 2; SPACE surrounding + 
%GTM-E-CMD, Command expected but not found
 set x=1 + 2
         ^-----
GTM>set x=1+2; no SPACE in expression
GTM>write x
3
GTM>set x= 9 ; SPACE after =  
%GTM-E-EXPR, Expression expected but not found
 set x= 9
       ^-----
GTM>set x=9; no SPACE after=
GTM>write x
9

Math expression
Comparing to other language, math operator has no order precedences, a expression is evaluated from left to right

GTM>write 1+2*3
9
GTM>write (1+2)*3
9

Parentheses has to be used to make thing work as expecting.

GTM>write 1+(2*3)
7