How to design a flexible Erlang protocol stack creation API

Question

Unsatisfied with my current approach I'm just trying to redesign the way I build protocol stacks in Erlang. The feature ordered by importance:

1. Performance

2. Flexibility and implementation speed adding new protocol variants

3. It would help development to explore the protocol variants from the shell

My current model (alreday described in this question) is getting at its limits besides the ugly asymmetry of send() by function call and receive by message.

The overall picture of the whole protocol engine looks like this:

Bottom part:

• There a several ports or maybe also sometimes a gen_tcp at the bottom of each stack (there are multiple identical stacks for independent channels so we can't be too static here just registering processes, have to pass Pids everywhere.

• On top of the ports are a few modules that are managed by supervisor (started with the system and in the absence of errors staying the whole lifetime).

Top part:

• Triggered by event occurrence (in the general sense not in the event_handler sense) are connection oriented protocol ends (e.g. with connect() and close() semantics.

• The top end of the protocol stack can probably only be started dynamically because the modules stacked on top of each other to form the stack are dynamically configurable and might change from connection to connection.

• Currently planned would be passing a List of module names + optional params from the toplevel that get consumed while connect() is called down the stack.

• Toplevel processes will be linked so when anything goes wrong here the whole connection fails.

Types of modules and types of communication between them

There are several kinds of modules found so far:

• Stateless filter modules

• Modules with state, some fit gen_server, some gen_fsm but most will probably be simple server loops since selective receive will be useful and simplify code quite often.

Types of communication between layers:

• Independent sending and receiving of packets (independent as seen from the outside)

• Synchronous calls that send something, block until there is a answer and then return the result as return value.

• Multiplexers that talk down to multiple modules (thats my definition here to ease discussion)

• Demultiplexers that have different attachment points (named by atoms currently) to talk with upward facing modules.

Currently my only demultiplexers are in the statical bottom part of the stack and not in the dynamically created top part. Multiplexers are only in the top part currently.

In the answers and comments of my linked previous question handling I heard that generally the API should only consist of functions and not messages and I concur with this unless convinced otherwise.

Please excuse the lengthy explanation of the problem but I think it is still of general use for all kinds of protocol implementation.

I will write what I have planned so far in the answers and also will also explain the resulting implementation and my experience with it later in order to achieve something generally useful here.

Answer 1

I'll throw in what I have planned so far as part of the answers:

• connect gets passed a List of modules to stack, looking like a proplist in case of params e.g:

  connect([module1, module2, {module3, [params3]}], param0, further_params)
  

  each layer strips off the head and calls the next layers connect.

• connect() "somehow" passes fun references up and/or down the layers

  • send for async sending down the stack will be returned by the lower level connect
  • recv for async receiving up the stack will be passed as param to the lower level connect
  • call for sync sending and waiting for an reply returned -- not sure how to handle these, probably also returned from the lower level connect

• Multiplexers routing lists might look like these

  connect([module1, multiplexer, [[m_a_1, m_a_2, {m_a_3, [param_a_3]}], 
                                  [m_b_1, m_b_2],
                                  [{m_c_1, [param_c_1]}, m_c_2]], param0, 
                                                                  further_params]).
  

---

Currently I decided there will not be a extra function for synchronous calls, I'm just using send for it.

There is implementation example of the idea in this case for a module which is stateless: encode/1 and decode/1 do some for and back transformation on packets e.g. parse binary representation into a record and back:

connect(Chan, [Down|Rest], [], Recv_fun) ->
    {Down_module, Param} = case Down of
                               {F, P} -> {F, P};
                               F when is_atom (F) -> {F, []}
                           end,
    Send_fun = Down_module:connect(Chan, Rest, Param,
                                   fun(Packet) -> recv(Packet, Recv_fun) end),
    {ok, fun(Packet) -> send(Packet, Send_fun) end}.

send(Packet, Send_fun) ->
    Send_fun(encode(Packet)).

recv(Packet, Recv_fun) ->
    Recv_fun(decode(Packet)).

As soon as I have a stateful example I'll post it too.