Slow Erlang Consumer

Erlang allows processes to send messages asynchronously. The receiving process queues all messages it gets until it matches them.= A common problem people eventually run in to is that a producer process sending messages to a consumer process produces messages faster than the consumer can handle. The slow erlang consumer problem can lead to horror stories of erlang ate all my memory. Consider when the producer is always faster than the consumer and it never stops. Then really you need to make more consumers and load balance. Maybe even using erlang's famous clustering abilities. Another possibility is that the producer may have spurts of activity where it just bogs down the consumer temporarily. This might be ok actually unless you start running out of memory. Its sometimes useful then to implement your own message queueing mechanism that can monitor the state of the messages and how many are piling up. If the pile of messages gets too big then you do something about it. Tell the producer process "hey stupid, slow down" or maybe just bust out the BFG9000 and take a big fat shot at that lousy producer. In any case this problem really isn't a problem until you have actually seen memory run out and in those cases it might just be easier to buy more of the stuff or find a real solution to the slow consumer like making it faster. A final possibility is that you have failed to tie together your producers and consumers so that when one one fails they both crash together. Failure in the consumer means a log jam of messages just pile up in its queue. Instead the consumer failure should result in a crash of itself and the producer. They should then be restarted through a supervisor in a semi-sane manner.