advertisement

Distributed Middleware Factory

19 %
81 %
advertisement
Information about Distributed Middleware Factory

Published on May 2, 2008

Author: paulrene

Source: slideshare.net

Description

How to do large scale distributed software development
advertisement

Distributed Middleware Factory Ramón Arellano (Accenture) and Paul René Jørgensen (Telenor)

About us Ramón Arellano Work experience 6 years of experience as a technical consultant from Accenture E-mail [email_address] Paul René Jørgensen Work experience 5 years of experience from Telenor E-mail [email_address]

Ramón Arellano

Work experience

6 years of experience as a technical consultant from Accenture

E-mail

[email_address]

Paul René Jørgensen

Work experience

5 years of experience from Telenor

E-mail

[email_address]

Distributed Middleware Factory - definition Distributed In the geographical sense of the word, i.e. various locations spread around the world Middleware The communication layer that lets applications interact across hardware and network environments Factory A facility used to manufacture goods in an effective way, using standardized industrial processes and specialized equipment

Distributed

In the geographical sense of the word, i.e. various locations spread around the world

Middleware

The communication layer that lets applications interact across hardware and network environments

Factory

A facility used to manufacture goods in an effective way, using standardized industrial processes and specialized equipment

Distributed Middleware Factory – definition continued Multiple teams in different locations, producing software for system integration, on the same platform, using standardized architecture and tools

Multiple teams in different locations, producing software for system integration, on the same platform, using standardized architecture and tools

So what are we going to talk about? Middleware platform for Telenor Fixed Line Services Metro - A distributed middleware factory In this talk we’ll share our experiences from its evolution and some secrets from the past 6 years

Middleware platform for Telenor Fixed Line Services

Metro - A distributed middleware factory

In this talk we’ll share our experiences from its evolution and some secrets from the past 6 years

Epochs in Metro history Epoch 0 – Medieval Age Epoch 1 – Gunpowder Age Epoch 2 – Enlightenment Epoch 3 – Industrial Age Epoch 4 – Modern Age

Epoch 0 – Medieval Age

Epoch 1 – Gunpowder Age

Epoch 2 – Enlightenment

Epoch 3 – Industrial Age

Epoch 4 – Modern Age

Epoch 0 – Medieval Age Year 2000 Challenges Where is the vision for the future? Use of the existing middleware platform is not mandatory Kings and kingdoms and their point to point integrations Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Year 2000

Challenges

Where is the vision for the future?

Use of the existing middleware platform is not mandatory

Kings and kingdoms and their point to point integrations

The Metro Vision All integration between systems for Telenor Fixed Line services to be implemented through Metro platform Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

All integration between systems for Telenor Fixed Line services to be implemented through Metro platform

Metro platform goals To offer test and production environments to all development project To enforce architecture and development standards To supply a common set of core components Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

To offer test and production environments to all development project

To enforce architecture and development standards

To supply a common set of core components

Distributed development model The platform is managed and controlled by a small, centralized team The Telenor employer selects software supplier The software supplier performs the implementation on the platform Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

The platform is managed and controlled by a small, centralized team

The Telenor employer selects software supplier

The software supplier performs the implementation on the platform

Distributed vs Centralized Employer chooses software supplier Development is done where the supplier is Easier to get new developers up and running Requires a stronger regime and documented standards Platform team chooses software supplier Development is done in-house Easier communication between all teams involved Does not scale so easily Platform Platform Employer Employer Employer Employer Employer Supplier Employer Employer Employer Employer Employer Supplier Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Employer chooses software supplier

Development is done where the supplier is

Easier to get new developers up and running

Requires a stronger regime and documented standards

Platform team chooses software supplier

Development is done in-house

Easier communication between all teams involved

Does not scale so easily

Epoch 1 – Gunpowder Age Year 2001 Challenges The volume and complexity of Telenor Common architecture How do we organize SI-testing? Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Year 2001

Challenges

The volume and complexity of Telenor

Common architecture

How do we organize SI-testing?

Statistics In an average hour the platform is processing 61 200 web requests from the internet 115 200 EJB invocations 2 880 000 database queries 65 MB log messages (1,5GB in a day) Volume Over 50 systems integrated Average 200 deployments a year Over 250 different developers from all over the world have written code Metro platform team guides up to 30 teams at the same time Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

In an average hour the platform is processing

61 200 web requests from the internet

115 200 EJB invocations

2 880 000 database queries

65 MB log messages (1,5GB in a day)

Volume

Over 50 systems integrated

Average 200 deployments a year

Over 250 different developers from all over the world have written code

Metro platform team guides up to 30 teams at the same time

Name a technology, and we’ve got it EBXML Tibco B2B SOAP, Proprietary HTTP/XML Web Services Proprietary XML (JItegra Java COM bridge) IIS / ASP J2EE, RMI, Corba Java Terminal, Screen Scraping, MQ, CICS IBM Mainframe (MVS) Siebel XML Siebel JDBC, ODBC, DB2Connect Oracle, Sybase, MySQL, DB2 Protocol Technology Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Technology at the base J2EE is the chosen technology IBM WebSphere is the chosen application server Started off with IBM WebSphere 3.5 and J2EE 1.1 WSAD as IDE (based on Eclipse) IBM MQ Server for messaging Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

J2EE is the chosen technology

IBM WebSphere is the chosen application server

Started off with IBM WebSphere 3.5 and J2EE 1.1

WSAD as IDE (based on Eclipse)

IBM MQ Server for messaging

Layered architecture Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Informal roles Experience has shown us the importance of having roles in a team The roles are more or less self imposed Examples The hardware geek The Maven guru The support girl The documentation dude The tool guy Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Experience has shown us the importance of having roles in a team

The roles are more or less self imposed

Examples

The hardware geek

The Maven guru

The support girl

The documentation dude

The tool guy

Test Environments Implementation, building and unit testing on developer’s laptop System integration testing on 60 blade servers running the full J2EE runtime environment One for each development project! Acceptance Test environment administered by the platform team Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Implementation, building and unit testing on developer’s laptop

System integration testing on 60 blade servers running the full J2EE runtime environment

One for each development project!

Acceptance Test environment administered by the platform team

Epoch 2 - Enlightenment Year 2002-2003 Challenges We want to be more innovative How do we get implementation projects to help develop the platform? How do we organize our work and control the deliverables? Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Year 2002-2003

Challenges

We want to be more innovative

How do we get implementation projects to help develop the platform?

How do we organize our work and control the deliverables?

The 80/20 rule Innovative and flexible leaders introduced the 80/20 rule Developers were allowed to use 20% of their time to learn and play with new technologies Highly motivating, and has undoubtedly been fruitful for Metro Technologies such as multicasting (Log), FishEye (CVS), were introduced by experimenting Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Innovative and flexible leaders introduced the 80/20 rule

Developers were allowed to use 20% of their time to learn and play with new technologies

Highly motivating, and has undoubtedly been fruitful for Metro

Technologies such as multicasting (Log), FishEye (CVS), were introduced by experimenting

Boiling soup on stone The platform team is responsible for setting the standards for many projects and developers Dave Thomas's book, “The Pragmatic Programmer”, talks about “boiling soup on stone” Introducing new technologies and standards by curiosity and envy The whole package more valuable than each part Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

The platform team is responsible for setting the standards for many projects and developers

Dave Thomas's book, “The Pragmatic Programmer”, talks about “boiling soup on stone”

Introducing new technologies and standards by curiosity and envy

The whole package more valuable than each part

Metro Roadmap Our common workflow and methodology for all projects A set of phases with corresponding deliverables Waterfall model (may jump backwards and iterate) Each project gets their own supervisor responsible for QA Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Our common workflow and methodology for all projects

A set of phases with corresponding deliverables

Waterfall model (may jump backwards and iterate)

Each project gets their own supervisor responsible for QA

Roadmap control in Jira We’ve got a tool to enforce Metro Roadmap Atlassian’s inexpensive, but excellent tool Jira provides workflow functionality Controls the flow and stores all document deliverables It’s the platform team’s communication channel with development projects Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

We’ve got a tool to enforce Metro Roadmap

Atlassian’s inexpensive, but excellent tool Jira provides workflow functionality

Controls the flow and stores all document deliverables

It’s the platform team’s communication channel with development projects

Epoch 3 – Industrial Age Year 2004-2005 Challenges Lack of documented standards New developers must get up and running fast We feel we could be more efficient in getting new applications out Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Year 2004-2005

Challenges

Lack of documented standards

New developers must get up and running fast

We feel we could be more efficient in getting new applications out

Document everything Roadmap workflow Architecture principles Development environment setup Naming standards Step-by-step guidelines Testing and debugging Building and deployment FAQ Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Roadmap workflow

Architecture principles

Development environment setup

Naming standards

Step-by-step guidelines

Testing and debugging

Building and deployment

FAQ

Documentation in Confluence Atlassian’s professional wiki, Confluence, for all documentation Wiki: easily updated, everybody gets the current document version Ported existing documents from MS Word Also used for blogging news and alerts through mailing lists and RSS feeds Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Atlassian’s professional wiki, Confluence, for all documentation

Wiki: easily updated, everybody gets the current document version

Ported existing documents from MS Word

Also used for blogging news and alerts through mailing lists and RSS feeds

We use Open Source Result of the 80/20 rule We’ve found free tools to improve our platform We love open source, and we give back Maven plugins Confluence plugins XRadar Jira plugins Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Result of the 80/20 rule

We’ve found free tools to improve our platform

We love open source, and we give back

Maven plugins

Confluence plugins

XRadar

Jira plugins

Leading Star Example Leading Star Driven development Improves application consistency TelenorShop – a fictitious application It shows live examples of all our common components and frameworks in use Source code for all Metro applications is openly available internally “Take a look at TelenorShop, and see how we’ve done it there” Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Leading Star Driven development

Improves application consistency

TelenorShop – a fictitious application

It shows live examples of all our common components and frameworks in use

Source code for all Metro applications is openly available internally

“Take a look at TelenorShop, and see how we’ve done it there”

Automation through 3rd party tools Automatic builds with unit testing using Maven Generating web-site with reports and JavaDoc using Maven Continuous integration using CruiseControl Mapping between data models using Dozer Support request through Jira Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Automatic builds with unit testing using Maven

Generating web-site with reports and JavaDoc using Maven

Continuous integration using CruiseControl

Mapping between data models using Dozer

Support request through Jira

Automation through self-made tools Applying for deployment slots to acceptance test environment Generating overview of dependencies between all J2EE applications on server Searching logs in all environments Updating application configuration Backing up WebSphere Application server configuration Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Applying for deployment slots to acceptance test environment

Generating overview of dependencies between all J2EE applications on server

Searching logs in all environments

Updating application configuration

Backing up WebSphere Application server configuration

Metro Certification A good factory scales easily New resources must be brought up to speed – fast No development without certification J2EE is a prerequisite Certification courses for Developers (2 day course) Project Leads (1 day course) 100 certified developers and project leads Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

A good factory scales easily

New resources must be brought up to speed – fast

No development without certification

J2EE is a prerequisite

Certification courses for

Developers (2 day course)

Project Leads (1 day course)

100 certified developers and project leads

Other certification benefits Efficient development Consistent solutions Stable solutions Reusable services Better developer attitude “ Ah, so that’s why we have to...” Externals pay a course fee ($) Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Efficient development

Consistent solutions

Stable solutions

Reusable services

Better developer attitude

“ Ah, so that’s why we have to...”

Externals pay a course fee ($)

Metro Certification content Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Epoch 4 – Modern Age Right now, right here Challenges Middleware is only noticed when something goes wrong How do we govern dependencies? Deployments are held back due to poor code quality Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Right now, right here

Challenges

Middleware is only noticed when something goes wrong

How do we govern dependencies?

Deployments are held back due to poor code quality

Commercialization We must learn how to better sell ourselves internally We want more applications on the platform and everyone to be aware of what Metro has to offer How Create a logo Offer introduction presentations Show off statistics of usage Present at JavaZone Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

We must learn how to better sell ourselves internally

We want more applications on the platform and everyone to be aware of what Metro has to offer

How

Create a logo

Offer introduction presentations

Show off statistics of usage

Present at JavaZone

Service Level Agreements SLAs between API owners (the kings) Wanted since the medieval age! No access to remote API without a signed SLA It will give us valuable information about inter-application dependencies We must work on enforcing this in the modern age! Requires a new tool (Any suggestions?) Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

SLAs between API owners (the kings)

Wanted since the medieval age!

No access to remote API without a signed SLA

It will give us valuable information about inter-application dependencies

We must work on enforcing this in the modern age!

Requires a new tool (Any suggestions?)

Code Quality Step-up the code QA work Make it easier for the reviewers to focus on code that is being altered Reviewers are notified immediately on CVS commit We’ll introduce Crucible into the review process Defines a workflow, creates an arena for dialog Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

Step-up the code QA work

Make it easier for the reviewers to focus on code that is being altered

Reviewers are notified immediately on CVS commit

We’ll introduce Crucible into the review process

Defines a workflow, creates an arena for dialog

Architects must write code “ Real insight comes from active coding” Practices of an Agile Programmer

“ Real insight comes from active coding”

Practices of an Agile Programmer

Summary Key tools Key lessons

Key tools

Key lessons

Key tools Confluence (wiki) for documentation No delay from writing to publication Jira No more forgotten issues in inboxes all over the place Issue management and process (roadmap) workflow Maven Stable and proven build system Eclipse based IDE Allow only one IDE Be an expert to support the developers

Confluence (wiki) for documentation

No delay from writing to publication

Jira

No more forgotten issues in inboxes all over the place

Issue management and process (roadmap) workflow

Maven

Stable and proven build system

Eclipse based IDE

Allow only one IDE

Be an expert to support the developers

Key Lessons Easy to understand workflow for everybody – Roadmap Documented standards and guidelines – Wiki And a way to teach them (Certification) Decision from the top : All integration between systems to be implemented through the platform

Easy to understand workflow for everybody – Roadmap

Documented standards and guidelines – Wiki

And a way to teach them (Certification)

Decision from the top : All integration between systems to be implemented through the platform

Any questions? paul-rene.jorgensen@telenor.com / ramon.arellano@accenture.com

Add a comment

Related pages

Middleware – Wikipedia

Middleware organisiert den Transport komplexer Daten (sog. messaging), ... Distributed Computing Environment (DCE) Enterprise Application Integration;
Read more

Distributed Systems/Middleware JMS - Politecnico di Milano

Politecnico di Milano Distributed Systems/Middleware: JMS 7 Queuing: Communication primitives Primitive Meaning Put Append a message to a specified queue
Read more

6.1 Introduction to CORBA 6.2 CORBA-IDL 6.3 Designing ...

Distributed Architecture 3. Middleware Systems 4. CORBA Architecture Chapter 6. ... Factory NamingContext EventChannel-Popular services include:
Read more

DistributedQueueBean - docs.oracle.com

Factory Methods: Factory methods in JMSBean; ... The messaging load is distributed across the topic members by pseudo-randomly accessing the distribution.
Read more

A Factory To Design and Build Tailorable and Verifiable ...

A Factory To Design and Build Tailorable and Verifiable Middleware Jérôme Hugues1, Fabrice Kordon2, Laurent Pautet1, and Thomas Vergnaud1 1 GET ...
Read more

Enterprise Integration Patterns (Middleware)---a Technical ...

Enterprise Integration Patterns (Middleware)---a Technical Reference Guide for Designing Mission-Critical Middleware Solutions
Read more

Architectural Patterns for Distributed Computing

Summary: As the technologies supporting design, development, and deployment of distributed applications continue to advance, it is more important than ever ...
Read more

Proactive Recovery in Distributed CORBA Applications

Proactive Recovery in Distributed CORBA Applications Soila Pertet and Priya Narasimhan Electrical & Computer Engineering Department Carnegie Mellon University
Read more

docs.oracle.com

iii Contents Preface
Read more