Download

Introduction to Untwisted

Event-driven programming is a programming paradigm in which the flow of a program is dependent of events. It is quite different from the usual approach of writing applications. Event-driven programming is sometimes tedious and hard to master although it is applicable to a wide range of scenaries and is important to understand. You will find event-driven techniques when parsing XML documments, writing network applications, writing GUI applications etc.

Hollywood Principle: "Don't call us. we'll call you."

Despite the humouristic side of this quote it is completely true. When writing applications on event-driven paradigm, we aren't going to have direct control over the program flow. It will mostly depend on events. There will exist handles that are called when these events occur.

Everyone has reasoned in the event-driven paradigm. In fact, we do it in our daily. Consider this situation; you go out for a quick walk then it starts raining. Your natural action is getting your umbrella. It is quite event-driven paradigm. You reacted to an event, you changed the flow of your life based on a simple event that is 'raining'.

The event driven approach can be ordinarily summarized as: EVENT -> HANDLE. It might sound trivial, in fact, it is trivial and simple but being creative in this paradigm isn't as easy as understanding this simple concept. There will exist other concepts related when writing real applications. There will be transformation of data along chains of events etc.

I got tired of dealing with event driven programming related to network applications in the usual way. I found some python frameworks quite messed up. It took me some months to abstract my own view of what the work of writing network applications in python should be. It was when i had the idea to write untwisted that is purely based on event-driven and dataflow concepts.

I can't tell you where untwisted differs from the most popular frameworks for writing asynchronous network applications in python. I can't even tell you which features untwisted has in advantage of others. But i can tell you something about untwisted, this thing is: It took me less time to figure out how to implement untwisted than learning any other framework in python out there for asynchronous network programming. I feel quite comfortable when implementing applications with untwisted. It is *my* feeling, it doesn't mean you will feel comfortable as well. This work is an attempt of passing ahead some ideas and concepts that i acquired along some time dealing with python and writing applications in it. This work has the humble goal of being helpful to someone else. I don't aim competing with any other python framework out there. I hope you enjoy this work as much as i enjoyed.

One effective way of learning some concepts it is going through an example where this concept is applied. I quite agree with that. So, i'm going to introduce a piece of code where untwisted plays its role. I will go further in event driven programming ahead using some diagrams and allusions.

is_up

from untwisted.network import *
from untwisted.utils.stdio import Client, CONNECT, CONNECT_ERR, lose
import errno

def on_connect(con):
    print 'Connected to google.'
    

def on_connect_err(con, err):
    print "It couldn't connect, errcode ", errno.errorcode[err]
    lose(con)


con = Spin()
Client(con)
xmap(con, CONNECT, on_connect)
xmap(con, CONNECT_ERR, on_connect_err)

con.connect_ex(('www.google.com.br', 80))

core.gear.mainloop()


The piece of code above might sound obvious to one who has experience with event-driven programming applied to construction of network applications although it might sound challenging to one who is new to this fascinating world. It's alright, things will become clearer and more complex along the way.

Leaving the pranks aside, the code just tries to connect to the site www.google.com.br if it goes fine it calls a handle named on_connect, if it fails to connect it calls a handle named on_connect_err. I guess it is not too hard to deduce what xmap, CONNECT, CONNECT_ERR means, right? You will not be able to entirely understand every piece of the code at first. You shouldn't worry with it for now. It is just a way of giving you an idea of how untwisted looks like.

There is below a small diagram that demonstrates the pattern for handles. This pattern pictures the most basic interaction between entities in a program using event-driven approach.


The dispatcher entity is responsible by calling the proper handles for the event being dispatched. It takes care of which handles should be called and which data these handles should receive.

This diagram is quite simple, it doesn't fully describe an event driven application flow. This diagram is gonna evolve in others more complicated diagrams. These diagrams are better explained ahead.


There will always exist an event loop in applications that use this model. This event loop is responsible by updating chain of events that are sent to the DISPATCHER object.

There might occur some situations where the events sent to the DISPATCHER object don't correspond to existing handles. It is there wasn't a mapping between a handle to an event, in this case, some implementations of DISPATCHERS might either discard the event or throw an exception. In untwisted the events sent to the DISPATCHER system which don't match a specific handle are just discarded.

There must exist a way to quit this event loop, some implementations just break the loop using an exception or just call a quit function. In untwisted it uses an exception to stop the event loop a.k.a reactor.

There is another diagram below exposing the interaction between the REACTOR a.k.a event loop of the application with the DISPATCHER object.

The REACTOR is responsible by updating the process. It takes control of the module's flow that sends events to the DISPATCHER object. In untwisted the reactor sends only two basic events to the DISPATCHER object. These two events are: WRITE, READ.

The diagram above is a simplified diagram for an application using an event-driven approach. It doesn't fully describe untwisted interactions either. Untwisted interactions will be exposed afterwards. My goal here is giving you a basic idea of how an event driven application works.

Before going ahead with diagrams for untwisted interactions i need to speak a bit about a dataflow diagram.

This describes the process of filtering input through filters. I implemented this technique in untwisted using an interesting approach. It is important to keep this diagram in mind to understand somethings afterwards. For now, just try to memorize this diagram and have in mind that it is a chain of pipes that transform data inputed and produce a product at the end of the chain of pipes. It is similar to when you do cat file | grep "pattern".

The diagram belo demonstrates a simplified model for untwisted interactions. This diagram doesn't fully explain untwisted interactions. I'm going through steps.

The reactor keeps track of all Spin instances. Spin instances are improved sockets. The reactor listens all time on the sockets. When one of them is ready for reading or writing then it dispatches the correspondent event to the Spin instance. It is either WRITE or READ events. It is not hard to notice that Spin instances in untwisted play the role of a dispatcher system. In fact, Spin instances inherit from a class named Mode that is a dispatcher system. Spin instances inherit from socket instance as well. It is interesting to think of Spin as a super socket class.

Before going deeper i will need to introduce a peculiar concept that is used to model stuff with untwisted. It is the concept of protocol. A protocol in untwisted is a set of handles that are mapped to events. These handles are instaled into a Spin instance. They work as consistent modules.

The ON-1, ON-2, ON-3 are just handles. Using this approach we have how to abstract stuff. We can put together a set of handles that are called on a given set of events. Very often handles from protocols will be operating on data that is defined in their classes. So, it is a way of them sharing data. We then have a consistent way of recording states among events. It is important to notice that a module might represent a protocol as well. It can have a set of functions defined inside that works cooperatively.

I need to expose a new diagram to picture one of untwisted's keys. It is shown below.

That diagram shows an important tactical feature that we have to handle internet protocols. It will be clearer ahead. The idea is basically: I can spawn new events from handles. These new events will fire/call other handles that can spawn new events as well and so on. This infinite chain of EVENT-A -> HANDLE-A -> EVENT-B -> HANDLE-B gives us a possibility of transforming the data that comes from a network connection a.k.a socket into other pieces of data that can be manipulated at the end of the chain of events. This chain of events work pretty much as a chain of pipes and filters. That is where i apply dataflow concepts to ease the job of handling internet protocols.

Events in untwisted are distinct of the data that they carry. Events in untwisted can be any kind of python object. You will understand better it afterwards. It is important trying to relate the diagrams above with the first example shown at the beginning of this howto.

I show below how this feature of spawning events from handles can be used to modularize applications by means of implementing abstractions of internet protocols using the concept of untwisted protocols.

This diagram describes basically the structure of the short example introduced at the beginning of this howto. The application is our set of handles that take action based on events generated by our untwisted protocols. The untwisted protocols have the job of transforming the raw data that comes from a connection into a product of events that can be advantaged in our application. Using this approach we will obtain a tree of events that carries data which is changed among handle calls.

Let us consider a real example where this model applies to. Suppose you want to write a simple irc bot that connects to an irc network and sits on a channel. When someone in that channel sends the sequence of characters 'foo' then the bot should reply to the channel with 'bar'. Irc networks send you informations on specific events. These informations comes in a codified way. They are raw text. You will need to transform this raw text into events in order to plug handles to these events. You will need to take care of other local events as well as READ and WRITE that corresponds to when you can read from the connection and when you can write to the connection.

The piece of raw text that we will be mostly interested is this. This piece of text comes from an irc connection when one sends a text to a channel. Here, the one's nick is hammond and the channel is #&philosophy.

hammond!~abner@unaffiliated/portrait PRIVMSG #&philosophy :foo

In order to implement this bot we will need to write a small and simple protocol to transform everything that comes from an irc network into something that we can advantage. We will not handle all cases but only this one in which a person sends a string of text to a channel that we specify in the bot.

"""
Name: ircbot.py
Description: Ircbot that connects to a network sits on a channel and
sends a string of text 'bar' when one of the channel users send a string of text 'foo'.
"""

# It imports Spin, xmap, and other useful functions.
from untwisted.network import *
# It imports the basic protocols and events.
from untwisted.utils.stdio import Client, Stdin, Stdout, CONNECT, CONNECT_ERR, CLOSE, LOAD, lose

# It imports a protocol to tokenizer data using a token.
from untwisted.utils.shrug import Shrug, FOUND

# socket lib.
from socket import socket, AF_INET, SOCK_STREAM

# A small implementation of a untwisted protocol for irc protocol.
from irc import Irc
import sys

def on_connect(con):
    # This protocol is responsible by spawning 
    # the event LOAD. It takes care of spawning a CLOSE
    # event when the connection is over.
    Stdout(con)
    
    # This protocol is responsible by installing a dump method
    # into the con instance. It takes care of sending everything
    # that goes through the dump method.
    Stdin(con)


    # This protocol is used to break the stream of data into chunks delimited
    # by '\r\n'. So, if the network sends 'data1\r\ndata2\r\ndata3\r\n' it will
    # spawns three times the event FOUND. It will carry 'data1', 'data2', 'data3'.
    Shrug(con)

    # This untwisted protocol is a tiny implementation of irc protocol.
    # It handles about 80% of the irc events. It is possible to be improved
    # and handle 100% of all irc events.
    Irc(con)

    # We want to print out on the screen all data that comes from the irc server.
    xmap(con, LOAD, lambda con, data: sys.stdout.write('%s\n' % data))

    # When the connection is over we need to destroy the Spin instance. The
    # lose function takes care of doing that for us.
    xmap(con, CLOSE, lambda con, err: lose(con))

    # When the event 'PRIVMSG' happens then we will have our handle
    # on_privmsg called. You could experiment other irc commands.
    xmap(con, 'PRIVMSG', on_privmsg)

    # When the irc server requires us to send back a PONG :server.
    xmap(con, 'PING', on_ping)

    # Our nick.
    con.dump('NICK %s\r\n' % NICK)
    
    con.dump('USER %s %s %s :%s\r\n' % (IRC_X, IRC_Y, IRC_Z, IRC_W))

    # Finally, it joins the channel.
    con.dump('JOIN %s\r\n' % CHANNEL)

def on_ping(con, prefix, addr):
    con.dump('PONG :%s\r\n' % addr)

def on_privmsg(con, prefix, argument):
    target, msg = argument.split(' :')
    
    # It checks whether the target is in fact a channel.
    # It might occur that someone else sent a msg to your bot.
    # In that case if i didn't do this check then the bot would
    # get in loop.
    if target.startswith(' #') and msg.startswith('foo'):
        con.dump('PRIVMSG %s :%s\r\n' % (target, MSG))

if __name__ == '__main__':
    # The ircbot nick.
    NICK = 'untwistedbot'
    # The irc server ip.
    ADDRESS = 'irc.freenode.org'
    # The irc server port.
    PORT = 6667
    # THe channel where the bot will sit on.
    CHANNEL = '#&math'
    
    IRC_X = 'user'
    IRC_Y = 'user'
    IRC_Z = 'user'
    IRC_W = 'user'
    
    # The msg sent when one sends 'foo'.
    MSG = 'bar'
    

    # It creates a socket as usual.
    # It will be used to send/receive data to the irc server.
    sock = socket(AF_INET, SOCK_STREAM)
    
    # We wrap the socket with a Spin instance.
    # When we wrap a socket with this instance
    # it automatically adds the socket to the reactor.
    # It sets the socket to non blocking mode as well.
    # From now on we will not be thinking imperatively.
    con = Spin(sock)

    # It attempts to connect to the network.
    # We have to call connect_ex cause the socket
    # is in blocking mode otherwise it would give us
    # an exception.
    con.connect_ex((ADDRESS, PORT))


    # This protocol spawns CONNECT or CONNECT_ERR.
    Client(con)

    # As i have installed the Client protocol which spawns
    # CONNECT or CONNECT_ERR we can link the on_connect handle
    # to the event CONNECT. When the CONNECT events is spawned
    # inside con then we will have our handle on_connect called.
    xmap(con, CONNECT, on_connect)
    
    # When there is no way of connecting to an ip then this
    # event will be spawned.
    xmap(con, CONNECT_ERR, lambda con, err: lose(con))

    # This is the reactor loop. When we call this method
    # untwisted reactor sits and starts listening on the
    # sockets which were created.
    core.gear.mainloop()

This is a straightforward example. It might sound complicated at first, but consider the following: this example can be extended to a more complicated one. It can turn into a decent and pretty funtional ircbot with just some changes. It is a matter of adding other handles for irc events.

"""
Name: irc.py
Description: A small abstraction for irc protocol.
"""


from untwisted.network import spawn, xmap
from untwisted.utils.shrug import Shrug, FOUND
from re import match, compile

RFC_STR = "^(:(?P<prefix>[^ ]+) +)?(?P<command>[^ ]+)( *(?P<argument> .+))?"
RFC_REG = compile(RFC_STR)

class Irc(object):
    def __init__(self, spin):
        xmap(spin, FOUND, self.split)
    
    def split(self, con, data):
        try:
            reg = match(RFC_REG, data)
            prefix = reg.group('prefix')
            command = reg.group('command')
            argument = reg.group('argument')
        except:
            pass
        else:
            command = command.upper()
            spawn(con, command, prefix, argument)


The code above is an implementation for irc protocol. It attempts to split the raw data coming from the irc connection into events carrying arguments. You can handle all irc events using this piece of code except the dcc ones. You can work the code out to make it more comfortable when handling events. You can split up a irc command arguments into chunks so you don't need to deal with it later as i did in the on_privmsg function.

This pattern for untwisted protocols should be followed in order to avoid unecessary complexity.

The code below is a simple example of implementing a server.

# Name: bar.py
# Description: It sets a server socket and prints everything 
# the clients send to the server.

from untwisted.network import xmap, Spin, core
from untwisted.utils.stdio import Server, Stdout, lose, ACCEPT, LOAD, CLOSE
from socket import *
import sys

def set_up_con(server, con):
    Stdout(con)
    xmap(con, CLOSE, lambda con, err: lose(con))
    xmap(con, LOAD, lambda con, data: sys.stdout.write('%s\n' % data))
    
sock = socket(AF_INET, SOCK_STREAM)
server = Spin(sock) 
server.bind(('', 1234))
server.listen(2)
Server(server)

xmap(server, ACCEPT, set_up_con)

core.gear.mainloop()

The code is straightforward, it sets up a server socket, installs the event handles and waits for untwisted reactor do the job of spawning the events. When one connects it installs a handle for the event LOAD, such an event carries data. The data is read out of the socket by the protocol Stdout. Stdout protocol once got data on its hand spawns LOAD. What would it happen if the clients disconnects? It is simple, Stdout or Stdin would spawn CLOSE, consequently it would call the lose function on the connection. The lose function takes care of removing the socket from untwisted reactor, so it will no more be processed.

The example below exemplifies another implementation of a server.

"""
Name: echo.py
Description: An echo server.
"""

# It imports basic objects.
from untwisted.network import *
from untwisted.utils.stdio import *

class EchoServer(object):
    """ A protocol abstraction for Echo protocol. """
    def __init__(self, spin):
        # It is basically spin.link.
        xmap(spin, ACCEPT, self.handle_accept)
       

    def handle_accept(self, server, client):
        """ When clients connect we are called. """

        # Install the basic protocol stdin to send data
        # asynchronously.
        Stdin(client)

        # Install the basic protocol stdout to receive
        # data asynchronously.
        Stdout(client)
       
        # The LOAD event is generated by Stdout
        # whenever some data is avaialable.
        xmap(client, LOAD, self.handle_load)

        # The CLOSE event is issued by either Stdin 
        # or Stdout protocol. It is when the host lost connection.
        xmap(client, CLOSE, self.handle_close)

    def handle_load(self, client, data):
        """ I dump back what it was written to me. """
        client.dump(data)

    def handle_close(self, client, err):
        """ I am called when the connection is lost. """
        # It tells untwisted to take this spin off the list
        # for reading, writting sockets.
        client.destroy()

        # Just closes the socket.
        client.close()

if __name__ == '__main__':
    # We create a Spin class pretty much as we would
    # do with a socket class.
    spin = Spin()
    spin.bind(('', 1234))
    spin.listen(200)


    # Install the Server protocol. This protocol is used
    # When we want to listen for incoming connections. 
    # It generates the ACCEPT event that happens
    # when some client connected.
    Server(spin)


    # Install the EchoServer application into spin.
    # Some applications might have a dual character
    # in the sense of spawning events through its circuit.
    # it isnt the case actually.
    EchoServer(spin)
    core.gear.mainloop()

This one is a typical echo server. It sits all time listening on a socket for new connections. When a client connects it accepts. When a client sends a string of text it echoes back through the connection to the client. The way of testing this example is: run echo.py with python echo.py on a terminal then connect to your ip on the port 1234 using a telnet. Once you are connected just send strings of text then you will notice the server echoing the strings which were sent back on your screen.

The example above shows how you can combine classes to build applications. It is sometimes useful to keep a state of the program, classes come in hand for that.

THe example below shows how you can build more complex servers.

"""
Name: funchat.py
Description: A simple chat server.

Usage:

    Run the funchat server in /demo/funchat/ with python funchat.py.

    You open two telnet sessions.

    Session 1.
    tau@spin:~/code/untwisted-code/demo/funchat$ telnet '0.0.0.0' 1235
    Trying 0.0.0.0...
    Connected to 0.0.0.0.
    Escape character is '^]'.
    Type a nick.
    Nick:euler
    hey gauss. i heard you are pretty good at maths.
    gauss:i heard you are good too.

    Session 2.
    tau@spin:~/code/untwisted-code/demo/funchat$ telnet '0.0.0.0' 1235
    Trying 0.0.0.0...
    Connected to 0.0.0.0.
    Escape character is '^]'.
    Type a nick.
    Nick:gauss
    euler:hey gauss. i heard you are pretty good at maths.
    i heard you are good too.
"""

# It imports basic objects.
from untwisted.network import *
from untwisted.utils.stdio import *
from untwisted.utils.shrug import *


class FunChat(object):
    def __init__(self, server):
        # It is basically spin.link.
        xmap(server, ACCEPT, self.handle_accept)
        self.pool = []

    def handle_accept(self, server, client):
        """ When clients connect we are called. """
        NICK_MSG = 'Type a nick.\r\nNick:'
        # Install the basic protocol stdin to send data
        # asynchronously.
        Stdin(client)

        # Install the basic protocol stdout to receive
        # data asynchronously.
        Stdout(client)

        # This protocol works on top of Stdout. 
        # It depends on Stdout events.
        # it generates the event FOUND
        # when it finds a token delimiter
        # in this case the delim is '\r\n'
        # So, whenever the client sends
        # 'This is a msg\r\nThat will split into two
        # lines\r\n' It issues FOUND twice 
        # carrying 'This is a msg' as argument in the first
        # time and 'That will split into two lines' as argument
        # in the second time.
        # Whenever LOAD happens Shrug appends
        # LOAD data argument to its internal buffer
        # in order to expect for a delim.
        Shrug(client, delim='\r\n')
        
        xmap(client, CLOSE, self.handle_close)

        client.dump(NICK_MSG)    
        
        # Wait the FOUND event to be issued again
        # in order to retrieve the nick chosen
        # and install the self.echo_msg handle
        # and add the client to the pool of clients.
        event, args = yield hold(client, FOUND)
        _, nick = args
        
        client.nick = nick 
        xmap(client, FOUND, self.echo_msg)

        self.pool.append(client)

    def echo_msg(self, client, data):
        """ This function echoes msgs through
            the clients connected to the server.
        """
        MSG_SHAPE = '%s:%s\r\n'

        for ind in self.pool:
            if not ind is client:
                ind.dump(MSG_SHAPE % (client.nick, data))

    def handle_close(self, client, err):
        """ I am called when the connection is lost. """

        # It tells untwisted to take this spin off the list
        # for reading, writting sockets.
        client.destroy()

        # Just closes the socket.
        client.close()

        # We no more will echo msgs to this client.
        self.pool.remove(client)

if __name__ == '__main__':
    # We create a Spin class pretty much as we would
    # do with a socket class.
    server = Spin()
    server.bind(('', 1234))
    server.listen(5)


    # Install the Server protocol. This protocol is used
    # When we want to listen for incoming connections. 
    # It generates the ACCEPT event that happens
    # when some client connected.
    Server(server)


    FunChat(server)


    # It processes the clients forever.
    core.gear.mainloop()

The idea is symmetrical to a simple echo server. It just waits for connections, asks for clients to pick up a nick. When the clients pick up their nicks it finally binds the clients. It shows the usage of hold which is a powerful tool sometimes when we don't want to lose some states inside a handle. It also lets the code clearer.

That is it. There are another examples in the untwisted tarball. Some functions may not have documentation, i plan to provide better docs soon. I hope it be useful to at least some of you. In case of it being useful to you in some way, please, let me know, you can email me at ioliveira@id.uff.br or just add me on msn.

Protocol implementations

uxirc an abstraction for irc protocol
http://uxirc.sourceforge.net/
uxrpc a loose implementation of a RPC protocol that supports amp.
http://uxrpc.sourceforge.net/

More complete examples

A simple web server
http://rapidserv.sourceforge.net
A nice IRC bot
http://ameliabot.sourceforge.net
An interface to play chess over fics
http://steinitz.sourceforge.net/
A fics bot
http://ficsbot.sourceforge.net/

Acknowledges

joo

Who helped me a lot giving some ideas and finding bugs and fixed all english grammar mistakes in this document. thank you joo, you are a nice virtual friend.