Tuesday, January 24, 2012

Fun with vim and IPython kernel

#First of all, get the development version of ipython

cd /home/philip/Packages
git clone https://github.com/ipython/ipython.git


# install this into your own virtualenv (I'm assuming at this point that you have created a virtualenv and that you have activated it).

pip install /home/philip/Packages/ipython

# do a 'which ipython' to check that it worked


# you will need development libraries for zeromq
sudo apt-get install libzmq-dev
pip install pyzmq

# check pyzmq was installed into the virtualenv

(py26)philip@desktop:~/git/project$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on desktop
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
>>> zmq.__path__
['/home/philip/git/project/py26/lib/python2.6/site-packages/zmq']


# get the vim-ipython plugin
pip install git+https://github.com/ivanov/vim-ipython.git


# start up an IPython kernel.
 ipython kernel

# and you get output like:
(py26)philip@desktop:~/git/project$ ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-32459.json


# startup vim and connect to the ipython kernel
gvim -c "IPython --existing kernel-32459.json"

or startup gvim and enter
:IPython --existing kernel-32459.json

In vim add this line:
3 + 4
Hit ctrl-s while still on this line and notice that a new window opens up with the result from the ipython kernel

now enter
a = 1

# from another shell
ipython qtconsole --existing kernel-32459.json

# to start up a qtconsole and connect it to the kernel
pip install pyside
ipython qtconsole --existing kernel-32459.json

do:
print a

And notice that the result of a is 1 because that is what we set a to from our vim session.


Dealing with Tuples in Erlang


Say in Python we have a function defined like this:
def foo(**kwargs):
    pass

In languages like Python, we can pass keyword arguments to a function like:
foo(a = 1, b = 2, c = 3, d = 1)



In Erlang the function may look like this:
foo(Args) -> ok.

A call to foo in Erlang would look like:
foo([{a, 1}, {b, 2}, {c, 3}, {d, 1}]).




In Python we could check if 'a' was passed in as a keyword argument with:
return 'a' in kwargs


Whereas in Erlang we need to do:
lists:keyfind('a', 1, Args)

Note that we are looking at the 1st element in our tuple (hence the 1 in the second argument and not a zero).

keyfind returns false if the key was not found, otherwise it returns the tuple




In Python, to find all keys with the value 1 we do:
[k for k, v in kwargs.items() if v == 1]


In Erlang we do:
lists:filter(fun({Key, Value}) -> Value == 1 end, Args).

and just in case you thought that
lists:keyfind(1, 2, Args).
would find all tuples with the second element equal to 1, it only returns the first tuple that it finds.










   


Tuesday, January 3, 2012

find command syntax

The find command is very powerful, but sometimes it trips me up when I forget to escape parenthesis, or add space inside parenthesis or forget to quote paths which contain an asterisk.

Here are some of the cases that get me most often:

 
# the -path option expects the full path:
find . -path "py26/*" -print     #does not work
find . -path "./py26/*" -print   #works !

# find files in either directory 
find . -path "./py26/*" -o -path "./.git/*" -print

# same as the previous example, but using parenthesis '()'
find . \( -path  "./py26/*" -o -path "./.git/*" \) -print

# the above command won't work if we leave out the spaces after the first parenthesis and before the last one.
find . \(-path  "./py26/*" -o -path "./.git/*"\) -print

# find all files except for files in py26 or .git
find . \( -path  "./py26/*" -o -path "./.git/*" \) -prune -o -print


Note, you don't need to add the -print at the end of the command (it is used by default, but shown here for clarity)



Friday, December 30, 2011

Getting further with Dialyzer

So in the previous post, I was able to compile modules in a way in which they could be analyzed with Dialyzer:

I installed erlang myself from source, so I set my ERL_TOP to where I built it from:
philip@desktop:~/s_server/src$ export ERL_TOP=~/Packages/otp




Now to built a PLT (Persistent Lookup Table).  I only include erlang applications which my application depends on:

philip@desktop:~/s_server/src$ dialyzer --build_plt -r . $ERL_TOP/lib/stdlib/ebin $ERL_TOP/lib/kernel/ebin


This took about 12 min for me using a quite old machine (P4 2.6 GHz).



Now I create my own PLT which is a combination of the previous PLT plus the PLT generated from my own code:

philip@desktop:~/s_server/src$ dialyzer --add_to_plt -r . --output_plt s_server.plt


Finally I can analyse my own code which is in my current directory:

philip@desktop:~/s_server/src$ dialyzer --plt s_server.plt -r .
  Checking whether the PLT s_server.plt is up-to-date... yes
  Proceeding with analysis...
s_server_tests.erl:14: The variable __V can never match since previous clauses completely covered the type 'true'
s_server_tests.erl:16: The variable __V can never match since previous clauses completely covered the type 'true'
s_server_tests.erl:48: The variable _ can never match since previous clauses completely covered the type 'false'
Unknown functions:
  eunit:test/1
 done in 0m1.17s
done (warnings were emitted)

The warnings which I received were in the eunit macros, and not in the actual code which I wanted to analyse.  It would be nice if there was a way to suppress these.

First steps with dialyzer

Today I tried to use dialyzer on my s_server module.  Don't  worry if you have not read any of my posts about this module, it doesn't do any thing useful !

At first I tried to analyse my test module:

philip@desktop:s_server/src$ erlc +debug_info s_server_tests.erl
philip@desktop:s_server/src$ dialyzer -c s_server_tests.erl --build_plt


Which gave me this result:

dialyzer: {dialyzer_error,"Byte code compiled with debug_info is needed to build the PLT"}
[{dialyzer_options,check_output_plt,1,
                   [{file,"dialyzer_options.erl"},{line,86}]},
 {dialyzer_options,postprocess_opts,1,
                   [{file,"dialyzer_options.erl"},{line,75}]},
 {dialyzer_options,build,1,[{file,"dialyzer_options.erl"},{line,63}]},
 {dialyzer_cl_parse,cl,1,[{file,"dialyzer_cl_parse.erl"},{line,218}]},
 {dialyzer_cl_parse,start,0,[{file,"dialyzer_cl_parse.erl"},{line,46}]},
 {dialyzer,plain_cl,0,[{file,"dialyzer.erl"},{line,60}]},
 {init,start_it,1,[]},
 {init,start_em,1,[]}]


The problem actually was that I should have given the beam files to dialyzer to analyse e.g.

philip@desktop:s_server/src$ dialyzer -c s_server_tests.beam --build_plt
  Creating PLT /home/philip/.dialyzer_plt ...
Unknown functions:
  eunit:test/1
  s_server:ping/0
  s_server:start_link/0
  s_server:stop/0
 done in 0m0.40s
done (passed successfully)

Wednesday, November 9, 2011

An Erlang Application in 5 minutes

Introduction
The purpose of this tutorial is to get an erlang application up and running with as little work as possible.  The application will consist of 1 supervisor which monitors a simple server (the appliation will be called s_server).

There will be 1 worker process with a gen_server behaviour.  When this process receives a ping, it will respond with a pong.  

Kind of like the "hello world" of Erlang applications !

Vim Setup
Yes, I'm going to use vim to speed things up, why break a 20 year old habbit ?

I installed  vim-erlang-skeletons from https://github.com/aerosol/vim-erlang-skeletons.git.  It gives you well documented complete skeletons of erlang behaviours.  If you don't want to use vim, then you can just google for a copy of the relevant skeleton.




Installing rebar
I'm still new to rebar, but it's definately a great tool to have to speed up writing erlang applications.  It's going to save us quite a bit of manual work here.

$ mkdir -p ~/Programming/Erlang
$ cd ~/Programming/Erlang
$ git clone https://github.com/basho/rebar.git
$ cd rebar && make


Create a new directory for the application
The application will be called s_server:
$ mkdir ~/Programming/Erlang/s_server
$ cd ~/Programming/Erlang/s_server

Then copy the rebar executable into the myapp project dir
$ cp ../rebar/rebar .


Creating an OTP App
$ ./rebar create-app appid=s_server

The app directory then looks like this:
s_server
|-- rebar
`-- src
    |-- s_server_app.erl
    |-- s_server.app.src
    `-- s_server_sup.erl


$ ./rebar compile


This creates the ebin directory with the compiled code as well as the application specification.

s_server
|-- ebin
|   |-- s_server.app
|   |-- s_server_app.beam
|   `-- s_server_sup.beam
|-- rebar
`-- src
    |-- s_server_app.erl
    |-- s_server.app.src
    `-- s_server_sup.erl


The app spec in ebin/myapp.app is created from the template in
s_server/src/s_server.app.src

You just finished making an OPT application !

Starting the Application
Start an erlang shell and add the ebin to it's path:
$ erl -pa ebin

From the erlang shell
1> application:start(s_server).

Now the application is running.

Of course you didn't write any worker process, so the application does nothing, but you can use appman to check that the application is running:
2> appman:start()

When you are done, leave the erlang shell
3> q().

Writing a worker process for the s_server application
Start up vim and in vim type :ErlServer to get the skeleton of a gen_server behaviour.

Enter :w src/s_server.erl to save it (and you can compile here also if you wish just to check that everything is still fine).

In the first line of code, set the correct module name
-module(s_server).

Also useful is to add under the module definition:
-compile([export_all, debug_info]).


In the API section, add the following code:

ping() ->
        gen_server:call(ping).


In the callbacks section, delete the existing handle_call skeleton and replace it with:
handle_call(ping, _From, State) ->
        Reply = pong,
        {reply, Reply, State}.

Getting the supervisor module ready
From within vim open up src/s_server_sup.erl.  Delete all existing code and type :ErlSupervisor to replace the current code with a better template.

In the init function, you need to replace AModule with the name of the module which contains our code for the gen_server i.e. s_server:
%s/AModule/s_server/g


Start the Application

Save the file and compile with:
$ ./rebar compile

Startup the erlang shell
$ erl -pa ebin

Start the s_server application
1> application:start(s_server).
ok

Now test it out:
2> s_server:ping().
pong

And to prove that the supervisor is restarting the server when it crashes, try to kill it:

First get the process number of the s_server from the output of regs().

3> regs().
** Registered procs on node nonode@nohost **
Name                  Pid          Initial Call                      Reds Msgs
:
s_server              <0.39.0>     s_server:init/1                     32    0

Then send the exit signal to it;
4> exit(pid(0, 39, 0), kill).

You can then check again from the regs(). command the process id of s_server.  This should now be different as the exit signal restarted it.


Thursday, October 13, 2011

where is mysql persisted to ?

quick answer:  The files are persisted to the directory specified by mysql's 'datadir' variable.

To get the current value do:

- start up mysql
mysql -u -p

- use the SHOW VARIABLES command
mysql> show variables;

or like this:

mysql> show variables like 'datadir';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| datadir       | /var/lib/mysql/ |
+---------------+-----------------+

If you cd into the datadir, here you can find there is on directory per database and inside of each database directory, each table has it's own .frm file

MySql5 uses innodb
http://en.wikipedia.org/wiki/InnoDB

as the storage engine which is creating these files.