advertisement

Building systems with Kamaelia

50 %
50 %
advertisement
Information about Building systems with Kamaelia

Published on October 7, 2007

Author: kamaelian

Source: slideshare.net

Description

This was a talk on how to build systems with Kamaelia given at Pycon UK. It
goes through from basics through to building a swarming P2P live radio
system.
advertisement

Kamaelia: Snap­Together Software Michael Sparks Senior Research Engineer BBC Research & Innovation Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

BBC R&I ...             PAL, Teletext, NICAM,   DVB,  DAB, Freeview, TV Anytime Majority of BBC Open Source Projects       inc Dirac, Kamaelia Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Key Research   goal? To make concurrency    easy to work with Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

How???   Single Reader/Single Writer Communicating Sequential Things Original context: scalable servers   Component Composition Written in Python Ideas are language agnostic Proof of concept in C++ Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

How???   No, I don't believe in “one    language to rule them all” *cough*Erlang*cough* Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  (demo) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  ie this: Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  What I'm aiming to get through   P2P Radio What? System Walkthrough Q&A Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

What?     Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  What if... ? Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

What has it been   used for?   Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Radio & Music  Interactive Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph              Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph Open Days ­ “this is what I need”            Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph Open Days ­ “this is what I need” Record All Radio PVR          Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph Open Days ­ “this is what I need” Record All Radio PVR Podcasts        Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph Open Days ­ “this is what I need” Record All Radio PVR Podcasts Proof of concept      Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Matt Biddulph Open Days ­ “this is what I need” Record All Radio PVR Podcasts Proof of concept Prototyping for experimentation  for new services Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Prototyping for  experimentation for new  services ... without any involvement from R&D     Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   Prototyping for  experimentation for new  services ... without any involvement from R&D due to being open source Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it?   BBC Macro Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Blog frontend Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/ & Meta Store Programme bbc one transcoder Kamaelia Backend bbc two transcoder DEMUXER bbc three transcoder TUNER bbc four transcoder cbbc transcoder cbeebies transcoder Kamaelia news 24 transcoder parliament transcoder

Who's used it?   Rapid Prototyping of Collaborative  Community Radio Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it? DJs in the Channel Controller Community Control Streaming Networ Server k Mixer Playout Web Front End Linked by Music Store videoconf Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Who's used it? DJs in the Channel Controller Community Control Streaming Networ Server k Mixer Playout Web Front End Linked by Music Store videoconf Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Google's Summer of  Code Student Work  2006, 2007 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Bit Torrent 2006   Programmes... WEBSITE Ingest BitTorrent Metadata Split Store Torrent Maker Web Client Torrent Peer           Swarm Store Torrent Peer Access Programmes... Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Open GL 2006   Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

2006 Trusted Communications   eg secure phone Decrypt ie trusted by the user Audio Decode Encrypt Decrypt BUFFER Encrypt Decrypt AudioPlayer Encrypt Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

2007   File­like Interface ­> (Much!) simpler usage outside  Kamaelia systems    IRC/AIM Components ­> causes creation of components ­> allows remote control of systems     Visual Component Generation ­> Sub component model ­> aim:easier creation of components  visually (rather than systems) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Other Systems Reframing for mobile, remuxing DVB­T  interactive aps for multicast streams,  Whiteboarding, VoIP testing, etc. Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Walkthrough     Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Simple Streamer     Pipeline( ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SingleServer( ), ).activate() Pipeline( TCPClient( host = quot;127.0.0.1quot;, port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Simple Streamer     Pipeline( ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SingleServer( ), ).activate() Pipeline( TCPClient( host = quot;127.0.0.1quot;, port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Simple Presentation Tool    Graphline( NEXT = Button(caption=quot;Nextquot;, msg=quot;NEXTquot;, position=(72,8)), PREV = Button(caption=quot;Previousquot;, msg=quot;PREVquot;, position=(8,8)), FIRST = Button(caption=quot;Firstquot;, msg=quot;FIRSTquot;, position=(256,8)), LAST = Button(caption=quot;Lastquot;, msg=quot;LASTquot;, position=(320,8)), CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), linkages = { (quot;NEXTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;PREVquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;FIRSTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;LASTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;CHOOSERquot;,quot;outboxquot;) : (quot;IMAGEquot;,quot;inboxquot;), } ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Simple Presentation Tool    Graphline( NEXT = Button(caption=quot;Nextquot;, msg=quot;NEXTquot;, position=(72,8)), PREV = Button(caption=quot;Previousquot;, msg=quot;PREVquot;, position=(8,8)), FIRST = Button(caption=quot;Firstquot;, msg=quot;FIRSTquot;, position=(256,8)), LAST = Button(caption=quot;Lastquot;, msg=quot;LASTquot;, position=(320,8)), CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), linkages = { (quot;NEXTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;PREVquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;FIRSTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;LASTquot;,quot;outboxquot;) : (quot;CHOOSERquot;,quot;inboxquot;), (quot;CHOOSERquot;,quot;outboxquot;) : (quot;IMAGEquot;,quot;inboxquot;), } ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Diving Inside? from Axon.ThreadedComponent import threadedcomponent class ConsoleReader(threadedcomponent): def main(self): eol = quot;nquot; while 1: # this blocks so we use a thread line = raw_input(quot;>>> quot;) line = line + eol self.send(line, quot;outboxquot;) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Diving Inside? from Axon.Component import component class ConsoleEchoer(component): def main(self): while 1: while self.dataReady(quot;inboxquot;): data = self.recv(quot;inboxquot;) _sys.stdout.write(str(data)) _sys.stdout.flush() yield 1 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Used thus... Pipeline( ConsoleReader(), ConsoleEchoer(), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Better Streamer     def UserProtocol(): return ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SimpleServer( protocol=UserProtocol, port=1601 ).activate() Pipeline( TCPClient( host = quot;127.0.0.1quot;, port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Better Streamer     def UserProtocol(): return ReadFileAdaptor( filename = 'video.drc', bitrate = 400000 ), SimpleServer( protocol=UserProtocol, port=1601 ).activate() Pipeline( TCPClient( host = quot;127.0.0.1quot;, port = 1601 ), DiracDecoder( ), MessageRateLimit( messages_per_second = 15, buffer = 15 ), VideoOverlay( ), ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Minimal Radio P2P Core    Backplane(“RADIO”).activate() Pipeline( TCPClient( host = quot;192.168.1.1quot;, port = 1601 ), PublishTo( “RADIO” ), ).activate() Pipeline( SubscribeTo(“RADIO”), UnixProcess(“mplayer - “), ).activate() def UserProtocol(): return SubscribeTo( “RADIO” ), SimpleServer( protocol=UserProtocol, port=1601 ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Minimal Radio P2P Core    Backplane(“RADIO”).activate() Pipeline( TCPClient( host = quot;192.168.1.1quot;, port = 1601 ), PublishTo( “RADIO” ), ).activate() Pipeline( SubscribeTo(“RADIO”), UnixProcess(“mplayer - “), ).activate() def UserProtocol(): return SubscribeTo( “RADIO” ), SimpleServer( protocol=UserProtocol, port=1601 ).run() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming P2P ­ idea      Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming P2P ­ setup      class SimpleSwarm(Axon.Component.component): clients = [] rr = 0 maxclients = 4 def __init__(self, port): super(SimpleSwarm, self).__init__() self.__class__.port = port Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming P2P ­ server      def main(self): while 1: while self.dataReady(quot;inboxquot;): data = self.recv(quot;inboxquot;) req = data.split(quot; quot;) if req[0] == quot;REQCONNECTquot;: if len(self.clients) < self.maxclients: reqfrom = req[1] self.clients.append(reqfrom) self.send( quot;CONNECT quot; + str(self.port) , quot;outboxquot; ) else: reqconn = self.clients[self.__class__.rr] self.__class__.rr = (self.__class__.rr+1) % self.maxclients self.send( quot;REQCONNECT quot; + reqconn ) if not self.anyReady(): self.pause() yield 1 Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming P2P ­ client      @staticmethod def clientRequest(rootip, rootport, myip, myport): servip, servport = rootip, rootport port = 0 while port == 0: l = LikeFile(TCPClient(servip, servport)) l.activate() l.send(quot;REQCONNECT %s:%dquot; % (myip, myport)) resp_raw = l.recv() resp = resp_raw.split(quot; quot;) if resp[0] == quot;CONNECTquot;: port = int(resp[1]) ip = servip elif resp[0] == quot;REQCONNECTquot;: servip, servport = resp[1].split(quot;:quot;) servport = int(servport) return servip, port Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

P2P Radio – Client (was)      Pipeline( TCPClient( host = quot;192.168.1.1quot;, port = 1601 ), PublishTo( “RADIO” ), ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming Radio – Client      Graphline(CONFIGURE= ConnectToSwarm(rootip, rootcontrolport, myip, mycontrolport), CLIENT = Carousel(mkTCPClient), PUBLISHTO = PublishTo(quot;RADIOquot;), linkages = { (quot;CONFIGUREquot;,quot;outboxquot;):(quot;CLIENTquot;, quot;nextquot;), (quot;CLIENTquot;, quot;outboxquot;) : (quot;PUBLISHTOquot;, quot;inboxquot;), } ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming Radio – Helpers      class ConnectToSwarm(Axon.Component.component): def __init__(self, rootip, rootport, myip, myport): super(ConnectToSwarm, self).__init__() self.rootip = rootip self.rootport = rootport self.myip = myip self.myport = myport def main(self): servip, servport = SimpleSwarm.clientRequest( self.rootip, self.rootport, self.myip, self.myport) yield 1 self.send( ( servip, servport ), quot;outboxquot;) self.pause() yield 1 def mkTCPClient(args): return TCPClient(*args) Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Swarming Radio – Source      freq = 682.166670 # Sutton Coldfields feparams = { quot;inversionquot; : dvb3.frontend.INVERSION_AUTO, quot;constellationquot; : dvb3.frontend.QAM_16, quot;coderate_HPquot; : dvb3.frontend.FEC_3_4, quot;coderate_LPquot; : dvb3.frontend.FEC_3_4, } Pipeline( DVB_Multiplex(freq, [6210], feparams), # RADIO ONE PublishTo(quot;RADIOquot;), ).activate() Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Kamaelia is already a highly   capable platform that has  delivered some interesting  tools Currently version 0.5.0 Version reflects where we want to be Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  However it also shows that  you can make  concurrency easier to work  with, even in a normal  language Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  You can build your own  version if you follow our  “mini axon” tutorial using  your language of choice   Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Suggestions: Take it, use it Build cool stuff with it Break it, Fix it Steal the ideas Clone in your own  language Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

  Thanks for listening :­) http://kamaelia.sf.net/ michael.sparks@bbc.co.uk Kamaelia Michael Sparks, michaels@rd.bbc.co.uk, http://kamaelia.sf.net/

Add a comment

Related pages

Kamaelia - Concurrency made useful, fun

Kamaelia - Concurrency made useful, fun In Kamaelia you build systems from simple components that talk to each other. This speeds development, massively ...
Read more

AnsweredQuestions - kamaelia.org

Kamaelia - making concurrency simpler in python - AnsweredQuestions ... Kamaelia doesn't _directly_ simplify the process of building a protocol using.
Read more

Kamelia: highly concurrent and network systems tamed - BBC

Kamaelia is a project aimed at building large scale online media delivery systems for the long term. Large scale media systems are naturally concurrent ...
Read more

[Python] ANN: Kamaelia 0.2.0 released! - Grokbase

(17 replies) Kamaelia 0.2.0 has been released! ... syntactic sugar to make building Kamaelia systems simpler. (Specifically Graphline and pipeline systems.
Read more

R&D White Paper - BBC

Kamaelia: Highly Concurrent and Network Systems Tamed Michael Sparks Abstract Kamaelia is a project aimed at building large scale online media delivery
Read more

Building Systems | LinkedIn

View 70159 Building Systems posts, presentations, experts, and more. Get the professional knowledge you need on LinkedIn.
Read more

Kamaelia / Mailing Lists - SourceForge

... * Kamaelia systems should be creatable, ... Also, by writing code for building this using Kamaelia (Compose is written using Kamaelia) ...
Read more

Answered one of my questions - Google Groups

Kamaelia doesn't directly give me access to a connected socket descriptor. ... These are in effect tutorials about building (basic) network systems
Read more