Slow Erlang Consumer

Erlang allows processes to send messages asynchronously. The receiving process queues all messages it gets until it matches them.= A common problem people eventually run in to is that a producer process sending messages to a consumer process produces messages faster than the consumer can handle. The slow erlang consumer problem can lead to horror stories of erlang ate all my memory. Consider when the producer is always faster than the consumer and it never stops. Then really you need to make more consumers and load balance. Maybe even using erlang's famous clustering abilities. Another possibility is that the producer may have spurts of activity where it just bogs down the consumer temporarily. This might be ok actually unless you start running out of memory. Its sometimes useful then to implement your own message queueing mechanism that can monitor the state of the messages and how many are piling up. If the pile of messages gets too big then you do something about it. Tell the producer process "hey stupid, slow down" or maybe just bust out the BFG9000 and take a big fat shot at that lousy producer. In any case this problem really isn't a problem until you have actually seen memory run out and in those cases it might just be easier to buy more of the stuff or find a real solution to the slow consumer like making it faster. A final possibility is that you have failed to tie together your producers and consumers so that when one one fails they both crash together. Failure in the consumer means a log jam of messages just pile up in its queue. Instead the consumer failure should result in a crash of itself and the producer. They should then be restarted through a supervisor in a semi-sane manner.

Starting Erlang Programs

Starting an erlang program for someone new can be confusing. Erlang does not work like C or python or java. It really is unique in the way that it requires you to start your program.

Lets start off like I did for the first time, the books are nice and often give examples like.

1> c(something).
{ok, something}
2> something:do().
"hello"
3>

Well thats nice and dandy, now how do I just do something like I do with most programs, you know...

$ ./something
  "hello"

Lets see what options erl has.

$ erl --help
Erlang R14B02 (erts-5.8.3) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)
1>

Ooops... where's help? What just happened? Turns out --help isn't used by erl.

Instead erl has built in man pages. Not unix man pages mind you.

$ erl -man erl

Here you can finally see a huge multitude of options. The one in particular we are interested in is '-s' or setting an init function. The '-s' option takes at least a module and optionally a function followed by optional arguments.

Another aspect of -s thats interesting to take note of is that you may have more than one -s argument given to erl!

As an example lets use a simple module.

-module(blah_me).
-export([start/0]).

start() ->
    io:format("started my program~n"),
    ok.

And then in your favorite shell (mine is zsh).

$ erlc blah_me.erl
$ erl -s blah_me

And viola, we can run erlang starting it with any module, function, and arguments we want. In fact you can run any number of functions this way.

Try for example.

$ erl -s blah_me -s blah_me

I often write small shell scripts to pass in lots of arguments and things to erl when developing a program. This saves a lot of time starting of many common things needed. At some point however its a good idea to make your program use OTP releases and all the benefits they provide.