Erlang: problems connecting to node

How did you start the nodes, did you use
erl -setcookie my_cookie

Sometimes it is common to write

erl -cookie my_cookie % wrong, use setcooke, not cookie

erl --setcookie my_cookie % wrong, use only 1 dash

erl -setcookie=my_cookie % wrong, do not use an equals sign

What does the shell think the cookie value is ?
Try
1> erlang:get_cookie().

Is the value returned the same as the value as you intended to use for the cookie ?  If not, check the common mistakes above.

net_adm:ping fails:

Did you specify the remote node name as an atom ?

e.g. try 'a@localhost' instead of a@localhost

eunit truncates terms when outputting diff

Apparently this is not eunit's fault, but rather something to do with the shell. There is a workaround which is to output to file instead of the shell:

Use eunit with surefire:
eunit:test(my_module, [{report,{eunit_surefire,[{dir,"."}]}}]).

this will create a file called TEST-something.xml which will print out the full terms if a test fails

If you are running eunit via dialyzer like this:


$> rebar eunit

Then you can create the xml files by adding this line to your rebar.config:

{eunit_opts, [verbose, {report,{eunit_surefire,[{dir,"."}]}}]}.

This will make the tests more verbose (remove the verbose option if you do not want that).

It will also create the xml files in the same directory as the beam files created for the test which is called .eunit.

For example if you have the following structure for your app:

| my_app

|-- src

|-- ebin

| -- .eunt

Compiling Erlang modules for Unit Test

%% When compiling erlang modules for testing, some of the common options are
  debug_info
and
  export_all
e.g. from the shell:
compile:file(my_module, [debug_info, export_all]).

%% or you can use
c(my_module, [debug_info, export_all]).

%% If my_module is not in the current path, you can specify the full path as a string:
c("apps/my_app/src/my_module.erl", [export_all, {outdir, "apps/my_module/ebin"}, {d, 'TEST'}]).

%% OR
c("src/bid_processor", [export_all, {outdir, "ebin"}, {d, 'TEST'}]).

%% If you suspect that your module under test does not behave according to the most recent code changes you can check that the shell has not loaded some stray beam file from another directory:
code:which(my_module).
%% This will return the full path to the module

%% And if you really want to you can unload the module with
code:purge(my_module).
code:delete(my_module).

%% When running eunit test cases it often happens that the diff of 2 terms is truncated. e.g. when you compare 2 large tuples.  You can output the test log to a file and in the file the terms are not truncated.

You can use
code:get_path().
to get a list of paths used in the erlang shell, most likely you will have to wrap that method in the shell command rp, otherwise the shell will truncate the output e.g.
rp(code:get_path()).


It is very useful if you have a directory called e.g. apps and which contains a collection of erlang applications to set the ERL_LIBS environment variable e.g.
export ERL_LIBS=/home/philip/erlang/apps
When you start up the erlang shell, and do code:get_path(), you will see that for each application, erlang has automatically set a path to the ebin directory.

erlang registering and monitoring global processes

Please note, the shell command numbers may not be in order as I put this together in a hurry.

% start up an erlang node
 erl -sname a -setcookie qwerty




%% create a new process which echos back anything it receives
(a@philipclarkeirl-desktop)1> Pid = spawn(fun() -> receive {Sender, Msg} -> Sender ! Msg end end).
<0.40.0>


%% test it out
(a@philipclarkeirl-desktop)2> Pid ! {self(), "Hello"}.
{<0.38.0>,"Hello"}

%% globally register the process
 (a@philipclarkeirl-desktop)3> global:register_name(myprocess, Pid).
yes

%% check it is registered
(a@philipclarkeirl-desktop)4> global:registered_names().
[myprocess]


(a@philipclarkeirl-desktop)6> global:whereis_name(myprocess) =:= Pid.
true
 
%% send to myprocess a message
(a@philipclarkeirl-desktop)7> global:send(myprocess, {self(), "Hello from shell"}).
<0.40.0>

%% note the return value is the pid found for myprocess
 
%% check in the shell that we received a response
(a@philipclarkeirl-desktop)8> flush().
Shell got "Hello from shell"
ok


%% start up another node
erl -sname b -setcookie qwerty

%% check that the process is globally registered
(b@philipclarkeirl-desktop)3> global:registered_names().
[myprocess]

%% monitor the process started in node a
(b@philipclarkeirl-desktop)11> MRef = erlang:monitor(process, global:whereis_name(myprocess)).
#Ref<0.0.0.83>



%% in node a, kill the process
(a@philipclarkeirl-desktop)33> exit(Pid, kill).
true


%% in node b
(b@philipclarkeirl-desktop)12> flush().
Shell got {'DOWN',#Ref<0.0.0.83>,process,<5799.85.0>,killed}
ok

Eunit terminology

Eunit Terminology

Based mostly on https://github.com/richcarl/eunit/blob/master/doc/overview.edoc

Simple Test

A function ending in _test and which takes no arguments

The test will always pass unless an exception is raised e.g. 1 =:= 2

You can use assert macros to cause the function to pass or fail

From the doc:

"A drawback of simple test functions is that you must write a separate

function (with a separate name) for each test case."

However you could write a _test() function which contains a list of assert statements ??

eunit:test(someModule) will export all simple tests.

Test

A fun expression that takes no arguments

e.g.

fun () -> ?assert(1 + 1 =:= 2) end.

Test Object

An assert which starts with an underscore 

"You can think of the initial underscore as signalling test object."

A simple test can be converted into a test object:

?_test(assert(BoolExpr)) % this is now a test object

Test Generator

A function ending in _test_

Returns a representation of a set of tests (that is a test set).

e.g.

basic_test_() ->
       fun () -> ?assert(1 + 1 =:= 2) end.

which is the same as

basic_test_() ->
       ?_assert(1 + 1 =:= 2).'''

Test generators cannot return simple tests ?

eunit:test(someModule) will export all test generators.

Simple Test Object (2nd definition)

A nullary functional value (i.e., a fun that takes zero arguments). 

e.g.

fun () -> ... end

Test Set

A list of test objects or a list of other test sets.

Instantiator

Used inside a fixture, takes some state and returns a test set

It behaves like a generator

Is defined by:

((R::any()) -> Tests)

It can be used in a setup fixture like this:

{setup, Setup, Cleanup, Tests | Instantiator}

or used in a foreach fixture like

{foreach, Setup, Cleanup, [Tests | Instantiator]}

 

Fixture

Sets up state for a test set and takes down the state at the end of the test set

By default a separate process is responsible for the setup and teardown and another process is used for the test. Therefore if the test causes a crash, the cleanup will always be done.

A setup fixture works differently to a foreach fixture.

A setup fixture sets up state, runs all test cases and then does a cleanup.

A foreach fixture will setup state for each test case and teardown after each test case.

e.g.

higher_order_test_() ->

    {"Higher Order Tests",

     [

      {"Setup Test",

       {setup,

fun () -> 4711 end,

fun (4711) -> ok end,

fun (X) ->

[{"1st", ?_assert(X =:= 4711)},

{"2nd", ?_assert(X =:= 4711)},

{"3rd", ?_assert(X =:= 4711)}]

end}

      },

      {"Foreach Test",

       {foreach,

fun () -> 4711 end,

fun (4711) -> ok end,

[fun (R) -> {"1st", ?_assert(R =:= 4711)} end,

fun (R) -> {"2nd", ?_assert(R =:= 4711)} end,

fun (R) -> {"3rd", ?_assert(R =:= 4711)} end]

       }

      }.

Fun with vim and IPython kernel

#First of all, get the development version of ipython

cd /home/philip/Packages

git clone https://github.com/ipython/ipython.git

# install this into your own virtualenv (I'm assuming at this point that you have created a virtualenv and that you have activated it).

pip install /home/philip/Packages/ipython

# do a 'which ipython' to check that it worked


# you will need development libraries for zeromq

sudo apt-get install libzmq-dev

pip install pyzmq

# check pyzmq was installed into the virtualenv

(py26)philip@desktop:~/git/project$ python

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on desktop
Type "help", "copyright", "credits" or "license" for more information.

>>> import zmq
>>> zmq.__path__
['/home/philip/git/project/py26/lib/python2.6/site-packages/zmq']

# get the vim-ipython plugin

pip install git+https://github.com/ivanov/vim-ipython.git

# start up an IPython kernel.

 ipython kernel

# and you get output like:
(py26)philip@desktop:~/git/project$ ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-32459.json


# startup vim and connect to the ipython kernel

gvim -c "IPython --existing kernel-32459.json"

or startup gvim and enter

:IPython --existing kernel-32459.json

In vim add this line:

3 + 4

Hit ctrl-s while still on this line and notice that a new window opens up with the result from the ipython kernel

now enter

a = 1

# from another shell

ipython qtconsole --existing kernel-32459.json

# to start up a qtconsole and connect it to the kernel

pip install pyside

ipython qtconsole --existing kernel-32459.json

do:

print a

And notice that the result of a is 1 because that is what we set a to from our vim session.

Dealing with Tuples in Erlang

Say in Python we have a function defined like this:

def foo(**kwargs):

    pass

In languages like Python, we can pass keyword arguments to a function like:

foo(a = 1, b = 2, c = 3, d = 1)

In Erlang the function may look like this:

foo(Args) -> ok.

A call to foo in Erlang would look like:

foo([{a, 1}, {b, 2}, {c, 3}, {d, 1}]).

In Python we could check if 'a' was passed in as a keyword argument with:

return 'a' in kwargs

Whereas in Erlang we need to do:

lists:keyfind('a', 1, Args)

Note that we are looking at the 1st element in our tuple (hence the 1 in the second argument and not a zero).

keyfind returns false if the key was not found, otherwise it returns the tuple

In Python, to find all keys with the value 1 we do:

[k for k, v in kwargs.items() if v == 1]

In Erlang we do:

lists:filter(fun({Key, Value}) -> Value == 1 end, Args).

and just in case you thought that

lists:keyfind(1, 2, Args).

would find all tuples with the second element equal to 1, it only returns the first tuple that it finds.

   

find command syntax

The find command is very powerful, but sometimes it trips me up when I forget to escape parenthesis, or add space inside parenthesis or forget to quote paths which contain an asterisk.

Here are some of the cases that get me most often:

 

# the -path option expects the full path:

find . -path "py26/*" -print     #does not work

find . -path "./py26/*" -print   #works !

# find files in either directory 

find . -path "./py26/*" -o -path "./.git/*" -print

# same as the previous example, but using parenthesis '()'

find . \( -path  "./py26/*" -o -path "./.git/*" \) -print

# the above command won't work if we leave out the spaces after the first parenthesis and before the last one.

find . \(-path  "./py26/*" -o -path "./.git/*"\) -print

# find all files except for files in py26 or .git

find . \( -path  "./py26/*" -o -path "./.git/*" \) -prune -o -print

Note, you don't need to add the -print at the end of the command (it is used by default, but shown here for clarity)

Getting further with Dialyzer

So in the previous post, I was able to compile modules in a way in which they could be analyzed with Dialyzer:

I installed erlang myself from source, so I set my ERL_TOP to where I built it from:
philip@desktop:~/s_server/src$ export ERL_TOP=~/Packages/otp




Now to built a PLT (Persistent Lookup Table).  I only include erlang applications which my application depends on:

philip@desktop:~/s_server/src$ dialyzer --build_plt -r . $ERL_TOP/lib/stdlib/ebin $ERL_TOP/lib/kernel/ebin

This took about 12 min for me using a quite old machine (P4 2.6 GHz).



Now I create my own PLT which is a combination of the previous PLT plus the PLT generated from my own code:

philip@desktop:~/s_server/src$ dialyzer --add_to_plt -r . --output_plt s_server.plt

Finally I can analyse my own code which is in my current directory:

philip@desktop:~/s_server/src$ dialyzer --plt s_server.plt -r .
  Checking whether the PLT s_server.plt is up-to-date... yes
  Proceeding with analysis...
s_server_tests.erl:14: The variable __V can never match since previous clauses completely covered the type 'true'
s_server_tests.erl:16: The variable __V can never match since previous clauses completely covered the type 'true'
s_server_tests.erl:48: The variable _ can never match since previous clauses completely covered the type 'false'
Unknown functions:
  eunit:test/1
 done in 0m1.17s
done (warnings were emitted)

The warnings which I received were in the eunit macros, and not in the actual code which I wanted to analyse.  It would be nice if there was a way to suppress these.

First steps with dialyzer

Today I tried to use dialyzer on my s_server module.  Don't  worry if you have not read any of my posts about this module, it doesn't do any thing useful !

At first I tried to analyse my test module:

philip@desktop:s_server/src$ erlc +debug_info s_server_tests.erl

philip@desktop:s_server/src$ dialyzer -c s_server_tests.erl --build_plt

Which gave me this result:

dialyzer: {dialyzer_error,"Byte code compiled with debug_info is needed to build the PLT"}
[{dialyzer_options,check_output_plt,1,
                   [{file,"dialyzer_options.erl"},{line,86}]},
 {dialyzer_options,postprocess_opts,1,
                   [{file,"dialyzer_options.erl"},{line,75}]},
 {dialyzer_options,build,1,[{file,"dialyzer_options.erl"},{line,63}]},
 {dialyzer_cl_parse,cl,1,[{file,"dialyzer_cl_parse.erl"},{line,218}]},
 {dialyzer_cl_parse,start,0,[{file,"dialyzer_cl_parse.erl"},{line,46}]},
 {dialyzer,plain_cl,0,[{file,"dialyzer.erl"},{line,60}]},
 {init,start_it,1,[]},
 {init,start_em,1,[]}]

The problem actually was that I should have given the beam files to dialyzer to analyse e.g.

philip@desktop:s_server/src$ dialyzer -c s_server_tests.beam --build_plt
  Creating PLT /home/philip/.dialyzer_plt ...
Unknown functions:
  eunit:test/1
  s_server:ping/0
  s_server:start_link/0
  s_server:stop/0
 done in 0m0.40s
done (passed successfully)

An Erlang Application in 5 minutes

Introduction
The purpose of this tutorial is to get an erlang application up and running with as little work as possible.  The application will consist of 1 supervisor which monitors a simple server (the appliation will be called s_server).

There will be 1 worker process with a gen_server behaviour.  When this process receives a ping, it will respond with a pong.  

Kind of like the "hello world" of Erlang applications !

Vim Setup
Yes, I'm going to use vim to speed things up, why break a 20 year old habbit ?

I installed  vim-erlang-skeletons from https://github.com/aerosol/vim-erlang-skeletons.git.  It gives you well documented complete skeletons of erlang behaviours.  If you don't want to use vim, then you can just google for a copy of the relevant skeleton.




Installing rebar
I'm still new to rebar, but it's definately a great tool to have to speed up writing erlang applications.  It's going to save us quite a bit of manual work here.

$ mkdir -p ~/Programming/Erlang

$ cd ~/Programming/Erlang

$ git clone https://github.com/basho/rebar.git

$ cd rebar && make

Create a new directory for the application
The application will be called s_server:

$ mkdir ~/Programming/Erlang/s_server

$ cd ~/Programming/Erlang/s_server

Then copy the rebar executable into the myapp project dir

$ cp ../rebar/rebar .

Creating an OTP App

$ ./rebar create-app appid=s_server

The app directory then looks like this:
s_server
|-- rebar
`-- src
    |-- s_server_app.erl
    |-- s_server.app.src
    `-- s_server_sup.erl

$ ./rebar compile

This creates the ebin directory with the compiled code as well as the application specification.

s_server
|-- ebin
|   |-- s_server.app
|   |-- s_server_app.beam
|   `-- s_server_sup.beam
|-- rebar
`-- src
    |-- s_server_app.erl
    |-- s_server.app.src
    `-- s_server_sup.erl


The app spec in ebin/myapp.app is created from the template in
s_server/src/s_server.app.src

You just finished making an OPT application !

Starting the Application
Start an erlang shell and add the ebin to it's path:
$ erl -pa ebin

From the erlang shell
1> application:start(s_server).

Now the application is running.

Of course you didn't write any worker process, so the application does nothing, but you can use appman to check that the application is running:
2> appman:start()

When you are done, leave the erlang shell
3> q().

Writing a worker process for the s_server application
Start up vim and in vim type :ErlServer to get the skeleton of a gen_server behaviour.

Enter :w src/s_server.erl to save it (and you can compile here also if you wish just to check that everything is still fine).

In the first line of code, set the correct module name

-module(s_server).

Also useful is to add under the module definition:
-compile([export_all, debug_info]).


In the API section, add the following code:

ping() ->
        gen_server:call(ping).


In the callbacks section, delete the existing handle_call skeleton and replace it with:
handle_call(ping, _From, State) ->
        Reply = pong,
        {reply, Reply, State}.

Getting the supervisor module ready
From within vim open up src/s_server_sup.erl.  Delete all existing code and type :ErlSupervisor to replace the current code with a better template.

In the init function, you need to replace AModule with the name of the module which contains our code for the gen_server i.e. s_server:
%s/AModule/s_server/g


Start the Application

Save the file and compile with:

$ ./rebar compile

Startup the erlang shell

$ erl -pa ebin

Start the s_server application
1> application:start(s_server).
ok

Now test it out:
2> s_server:ping().
pong

And to prove that the supervisor is restarting the server when it crashes, try to kill it:

First get the process number of the s_server from the output of regs().

3> regs().
** Registered procs on node nonode@nohost **
Name                  Pid          Initial Call                      Reds Msgs
:
s_server              <0.39.0>     s_server:init/1                     32    0

Then send the exit signal to it;
4> exit(pid(0, 39, 0), kill).

You can then check again from the regs(). command the process id of s_server.  This should now be different as the exit signal restarted it.

where is mysql persisted to ?

quick answer:  The files are persisted to the directory specified by mysql's 'datadir' variable.

To get the current value do:

- start up mysql
mysql -u -p

- use the SHOW VARIABLES command
mysql> show variables;

or like this:

mysql> show variables like 'datadir';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| datadir       | /var/lib/mysql/ |
+---------------+-----------------+

If you cd into the datadir, here you can find there is on directory per database and inside of each database directory, each table has it's own .frm file

MySql5 uses innodb
http://en.wikipedia.org/wiki/InnoDB

as the storage engine which is creating these files.

erlang how to fix "Can't set long node name"

For this post, my platform is Ubuntu 10.10

# Start up an erlang node with a long name as follows:
erl -name mynode


# This results in a long stack trace from erlang and the first line says:
{error_logger,{{2011,9,16},{18,1,5}},"Can't set long node name!\nPlease check your configuration\n",[]}

# The problem is the way your hostname is set. e.g.

philip@myserver:$ hostname
myserver

# now you can do
sudo hostname myserver.mydomainname.com

#next time you enter hostname you should get
myserver.mydomainname.com


# In order to make this permanent, you need to edit your /etc/hostname file and change the hostname from
myserver

# to be something like
myserver.mydomainname.com



# and now you can start the erlang node
philip@myserver:$ erl -name mynode
Erlang R13B03 (erts-5.7.4) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
(mynode@myserver.mydomainname.com)1>

Django South: Changing a field from null = True to null = False

# We have this table
mysql> show columns from myapp_mymodel where Field = "my_field_id";

+-------------+---------+------+-----+---------+-------+
| Field       | Type    | Null | Key | Default | Extra |
+-------------+---------+------+-----+---------+-------+
| my_field_id | int(11) | YES  | MUL | NULL    |       |
+-------------+---------+------+-----+---------+-------+
1 row in set (0.00 sec)


# We want to change this table so that column Null is NO for Field my_field_id


# First create an empty migration
bin/django schemamigration myapp mymodel_my_field_cannot_be_null --empty



#Then add the forward migration:
    def forwards(self, orm):
        #db.alter_column(table_name, column_name, field, explicit_name=True)
        db.alter_column('myapp_mymodel', 'my_field_id', models.ForeignKey(orm['myapp.MyModel'], null = False), explicit_name=True)


When this is run you get the following code executed:
DEBUG:django.db.backends:(0.000) SET FOREIGN_KEY_CHECKS=0;; args=()
DEBUG:south:south execute "
            SELECT kc.constraint_name, kc.column_name
            FROM information_schema.key_column_usage AS kc
            JOIN information_schema.table_constraints AS c ON
                kc.table_schema = c.table_schema AND
                kc.table_name = c.table_name AND
                kc.constraint_name = c.constraint_name
            WHERE
                kc.table_schema = %s AND
                kc.table_catalog IS NULL AND
                kc.table_name = %s AND
                c.constraint_type = %s
        " with params "['django', 'myapp_mymodel', 'FOREIGN KEY']"
DEBUG:django.db.backends:(0.111)
            SELECT kc.constraint_name, kc.column_name
            FROM information_schema.key_column_usage AS kc
            JOIN information_schema.table_constraints AS c ON
                kc.table_schema = c.table_schema AND
                kc.table_name = c.table_name AND
                kc.constraint_name = c.constraint_name
            WHERE
                kc.table_schema = django AND
                kc.table_catalog IS NULL AND
                kc.table_name = myapp_mymodel AND
                c.constraint_type = FOREIGN KEY
        ; args=['django', 'myapp_mymodel', 'FOREIGN KEY']
DEBUG:south:south execute "ALTER TABLE `myapp_mymodel` ;" with params "[]"
DEBUG:django.db.backends:(0.000) ALTER TABLE `myapp_mymodel` ;; args=[]
DEBUG:south:south execute "ALTER TABLE `myapp_mymodel` MODIFY `my_field_id` integer NOT NULL;;" with params "[]"
DEBUG:django.db.backends:(0.075) ALTER TABLE `myapp_mymodel` MODIFY `my_field_id` integer NOT NULL;;; args=[]
DEBUG:south:south execute "ALTER TABLE `myapp_mymodel` ALTER COLUMN `my_field_id` DROP DEFAULT;" with params "[]"
DEBUG:django.db.backends:(0.067) ALTER TABLE `myapp_mymodel` ALTER COLUMN `my_field_id` DROP DEFAULT;; args=[]
DEBUG:south:south execute "ALTER TABLE `myapp_mymodel` ADD CONSTRAINT `my_field_id_refs_id_15e652d5` FOREIGN KEY (`my_field_id`) REFERENCES `myapp_my_field` (`id`);" with params "[]"
DEBUG:django.db.backends:(0.283) ALTER TABLE `myapp_mymodel` ADD CONSTRAINT `my_field_id_refs_id_15e652d5` FOREIGN KEY (`my_field_id`) REFERENCES `myapp_my_field` (`id`);; args=[]
DEBUG:south:south execute "SET FOREIGN_KEY_CHECKS=1;" with params "[]"
DEBUG:django.db.backends:(0.000) SET FOREIGN_KEY_CHECKS=1;; args=[]
DEBUG:django.db.backends:(0.000) SELECT `south_migrationhistory`.`id`, `south_migrationhistory`.`app_name`, `south_migrationhistory`.`migration`, `south_migrationhistory`.`applied` FROM `south_migrationhistory` WHERE (`south_migrationhistory`.`app_name` = myapp  AND `south_migrationhistory`.`migration` = 0006_mymodel_my_field_cannot_be_null ); args=('myapp', '0006_mymodel_my_field_cannot_be_null')
DEBUG:django.db.backends:(0.000) INSERT INTO `south_migrationhistory` (`app_name`, `migration`, `applied`) VALUES (myapp, 0006_mymodel_my_field_cannot_be_null, 2011-09-16 14:37:50); args=('myapp', '0006_mymodel_my_field_cannot_be_null', u'2011-09-16 14:37:50')



# and the backward migraion is
    def backwards(self, orm):
        db.alter_column('myapp_mymodel', 'my_field_id', models.ForeignKey(orm['myapp.MyModel'], null = True), explicit_name=True)

Running the Thrift Tutorial with Python

See the previous post about installing thrift.


We will assume that you are currently in the thrift directory.
The tutorial.thrift file is written very well with lots of useful comments, but here are the few lines of code in this file which we really need:

namespace cpp tutorial
namespace java tutorial
namespace php tutorial
namespace perl tutorial

enum Operation {
  ADD = 1,
  SUBTRACT = 2,
  MULTIPLY = 3,
  DIVIDE = 4
}
struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}


exception InvalidOperation {
  1: i32 what,
  2: string why
}

service Calculator extends shared.SharedService {

   void ping(),
   i32 add(1:i32 num1, 2:i32 num2),
   i32 calculate(1:i32 logid, 2:Work w) throws (1:InvalidOperation ouch),
   oneway void zip()

}

Copy the files tutorial.thrift and shared.thrift into a new directory 

Generate the python files

thrift -r --gen py:new_style tutorial.thrift

where

-r makes thrift generate included files

--gen py:new_style makes thrift use the py generator with the optional argument of new_style to generate new style classes

Looking at gen-py/tutorial/Calculator.py you can see that the Calculator Service has been turned into an interface (class Iface).

The Iface class is inherited by class Client and class Processor

The Client implements the methods defined by the interface.

e.g. the add method does 2 things, calls send_add and returns recv_add.

The send_add method will write the name of the method 'add' and its arguments to the Thrift transport.

The recv_add method will receive the result of the add method.

In addition to the Client, there is a class called Processor who also implements Iface.

In the case of the add method, the Processor will read the arguments, call the handler who does the add operation and writes the result.

The handler is the important part as this is that class which you will write which implements the methods.

Installing Thrift

I'm doing this on ubuntu 10.10

First the following dependencies need installed:

sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libevent-dev automake libtool flex bison pkg-config g++

While that is going on you can download the latest thrift:
svn co http://svn.apache.org/repos/asf/thrift/trunk thrift

This creates the directory called thrift which you should cd into.

Next run
./bootstrap.sh
./configure
make
sudo make install

In the next post, I will look at running the tutorial which is found in the tutorial directory.

Python: Different types of member methods

In Python classes we can have the following methods:

1. Instance Methods
2. Static Methods
3. Class Methods

- Instance Methods
=====================
These are methods which are invoked upon an instance (object) of the class.
For example:

class MyClass:
    def myInstanceMethod(*args):
 print "inside myInstanceMethod, the arguments are:"
        for v in args:
            print v

Using this code we get:

>>>m = MyClass()
>>>m.myInstanceMethod()
    inside myInstanceMethod, the arguments are:
    <__main__.MyClass instance at 0xb7395a0c>

Which shows us that the object instance (commonly referred to as 'self') was passed into the instance method

If we try to call an instance method without an instance, we get:

>>>MyClass.myInstanceMethod ()
    TypeError: unbound method myInstanceMethod() must be called with MyClass instance as first argument (got nothing instead)

- Static Methods
=====================
In C++, static methods are usually used to fetch private static data from a class. The most common example is a static member which is incremented each time the constructor is called. This static member will then record the total number of instantations of the class.
In Python, the static method looks like this:

class MyClass:
    @staticmethod
    def myStaticMethod(*args):
 print "inside myStaticMethod, the number of arguments is ", len(args)
        for v in args:
            print v

Using this code we get:

>>>m = MyClass()
>>>m.myStaticMethod ()
     inside myStaticMethod, the number of arguments is  0
>>>MyClass.myStaticMethod ()
     inside myStaticMethod, the number of arguments is  0

The main difference in Python between the instance and static methods is that the static methods can be called by both instances and the class itself

- Class Methods
=====================

class MyClass:
    @classmethod
    def myClassMethod(*args):
        print "inside myClassMethod, the number of arguments is ", len(args)
        for v in args:
            print v

Using this code we get:

>>>m = MyClass ()
>>>m.myClassMethod ()
inside myClassMethod, the number of arguments is  1
__main__.MyClass

>>>MyClass.myClassMethod ()
inside myClassMethod, the number of arguments is  1
__main__.MyClass

Here we see (like the instance methods) we receive 1 argument, unlike the instance methods, the argument is the type of class, in the instance methods we received an object which was bound to the class.

Python: How useful are lambda functions ?

Lets say you have this function defined:

def square1(x) : return x*x

We can check what square is:

>>> type(square1)
 

And use it like this:

>>>square1 (8)
64

It can be rewritten as an anonymous function or lamda as:

square2 = lamda x: x*x

and again when we check the type we get

>>> type(square2)
 

And it works the same way as the previous function

>>>square2 (8)
64

Now say we define a list:

list = [1,2,3,4,5,6,7,8]

and then we want to get the square of each number. Either of these will work:

>>> map (square1, list)
[1, 4, 9, 16, 25, 36, 49, 64]

and same result with

>>> map (square2, list)
[1, 4, 9, 16, 25, 36, 49, 64]

So far the benefit of leaning lambda functions is small. We have only used them to write a small function on the fly and the function has to be small as lambdas do not support multiple statements.

Perhaps the main motivation for lambdas can be illustrated with the following:

strategy = {
    'square' : lambda x: x*x,
    'double' : lambda x: x*2,
    'half'   : lambda x: x/2,
}
list = [1,2,3,4,5,6,7,8]
while(True):
    try:
        x =  raw_input("What would you like to do with the list ? ")
        map(strategy[x], list)
    except KeyError, e:
        print "The value entered is not valid: " + str(e)

The dict called strategy contains 3 lambda functions and if we had not used lambdas the we would have had to add the following lines:

def square(x) : return x*x
def double(x) : return x*2
def half(x)   : return x/x

If our strategy dict contained a lot more functions, it would start getting tedious to define all of these and then since the definition of the function would be in another part of the code, it would not be as quick and easy to see what 'square' does for example.

Preventing class D from being derived from class B

In some cases there are classes who should not have children.
There are some ways to recognise this:

1) Are there any virtual methods ?
It is unlikely that you should derive from a class which has no virtual methods.  For example if the base class B has method foo which is non virtual, the derived class D cannot override (re-implement) or overload the method foo without hiding the method B::foo.

If you need to hide B::foo in class D, then it implies that class D is not really a B (violation of the D is-a B principle).


2) Is the destructor virtual ?
If a class D is derived from B then you have to assume that someone will quite legally do this:
B* pb = new D;
which implies that they will also want to do:
delete(pb);

Now if B does not have a virtual destructor, the destructor for B will be called and not the destructor for D.  This can lead to memory leakage if D has pointers to other objects on the heap.

Therefore if the destructor is not virtual, you have to assume that you should not be deriving from this class
In the book "C++ Coding Standards: 101 Rules, Guidelines, and Best Practices", it states
      "Make base class destructors public and virtual, or protected and nonvirtual"

If this is done it will help a user to see if it is intended to derive from a bass class

Discussion of std::vector

Note that in stl_vector.h, the destructor is defined as public and non-virtual.

Since the class has no virtual methods, then it is not a good idea to declare the destructor as virtual as this will add a virtual table lookup overhead for all method calls in this class. 
Also as the class is used as a concrete class the destructor has to be public.
Therefore in this case the destructor will be public and non-virtual.

The fact that the destructor is non-virtual and that there are no virtual methods implies that std::vector is not intended to be a base class of a derived class.

This may explain why there is no good documentation for deriving from stl containers, however I may make a few future posts where I do this and examine any pitfalls.

When to use friend functions over member methods

There are several arguments both for and against friend methods. Some of these arguments hold true in all cases and some only hold true depending on the context of programmer and his environment. Here is a summary of how I determine when to use friend functions or member methods.

1) Use member methods when it makes it much easier to see what methods are available
If you work in an organisation and you use an IDE where the member methods of each class are obvious, but the friend functions of the class are not obvious, use member functions. I've seen arguments on the web that say you should use non member friend functions to enforce encapsulation of the class and that crappy IDEs should not affect your decision. However in the commercial world where your company has invested in such and IDE, you can't afford to hide functionality from your colleagues.


2) If first arg cannot be a pointer to 'this', then a friend function will be easier to use 
For example consider operator<<

Say we have a class C which contains an int (e.g. 5) and a char (e.g. 'X') and we want to override operator<< to produce the result :5-X

int main() {
    C obj(5,'X');
    std::cout << "The value representing obj is " << obj << std::endl; // obj should be represented by 5-X

    return 0;
} 

If we define the class C with a the friend function operator<<, it should look like this:

class C { 
    friend std::ostream& operator<< (std::ostream&, const C&);
    public:
        C(int i, char c) : foo(i), bar(c)  {}; 
    private:
        int foo;
        char bar;

};
std::ostream& operator<< (std::ostream& ostr, const C& c) {
    ostr << c.foo << "-" << c.bar;
}

and the output is

The value representing obj is 5-X

Having the friend operator<< defined like this means we have the logical easy to read syntax of

std::cout << "The value representing obj is " << obj << std::endl;

We could have also used this syntax

std::cout << "The value representing obj is "; operator<<(std::cout, obj) << std::endl;

In the above, we are just more explicit in defining the arguments for operator<< Now say instead that we define operator<< as a member method:

class C {
    public:
        C(int i, char c) : foo(i), bar(c)  {};
        std::ostream& operator<< (std::ostream&);
    private:
        int foo;
        char bar;

};
std::ostream& C::operator<< (std::ostream& ostr) {
    ostr << foo << "-" << bar;
}

The only way to call this is

std::cout << "The value representing obj is "; obj.operator<<(std::cout) << std::endl;

Therefore defining operator<< to be a friend function is a much better option to using member methods. Some useful links:

http://www.faqs.org/faqs/alt.comp.lang.learn.c-c++/C++_FAQ_%28part_05_of_11%29/

read chapter [14.5] Should my class declare a member function or a friend function?

Also of interest
http://www.faqs.org/faqs/alt.comp.lang.learn.c-c++/C++_FAQ_(part_05_of_11)/#ixzz0cyteOeXg