Internet peering, with annotations

50 %
50 %
Information about Internet peering, with annotations

Published on March 14, 2014

Author: Brough



The Internet backbone consists of just over 6000 independent networks that exchange traffic in fashions that are not well understood outside of the backbone networking community. We explain how it works, how it has evolved and how it is continuing to evolve today.

This is a revised and annotated version of material most recently given as an invited presentation at OFC 2014, the optical fiber conference in San Francisco, in March 2014.

To provide higher resolution, I've also uploaded a version w/o annotations, i.e. just the graphics.

Why  care?       Peering  and  Transit  are  li3le  understood  and  yet  this  is  the  heart  of  Internet   infrastructure  economics  and  it  is  the  growth  of  Internet  infrastructure  that  ul:mately   funds  most  of  us  here  at  OFC.     I  going  to  describe  how  thousands  of  independent  organiza:ons  compete  but  also   exchange  traffic  and  con:nually  grow  an  ever  more  distributed  Internet   infrastructure.   1  

There  are  more  than  3  billion  Internet  users  and,  with  the  advent  of  very  low  cost   Android  smart  phones  ($24  in  India  in  Feb  2014),  it’s  likely  that  number  will  double  in   just  a  few  years.    There  are  also  tens  of  millions  of  local  networks  (perhaps  more  than   100  million  based  just  on  the  number  of  WiFi  routers  that  have  been  sold).    They   connect  to  the  Internet  through  tens  of  thousands  of  ISPs  which  may  be  classified  as   Access,  Aggrega:on  or  Backbone,  or  as  Local,  Regional,  Na:onal  or  Interna:onal  but,   of  course,  many  ISPs  cross  these  boundaries.       For  today,  I’m  going  to  focus  on  the  6000  or  so  major  ISPs  that  form  today’s  Internet   backbone.       But  to  understand  today’s  complex  environment,  it’s  useful  to  see  how  it  emerged.   2  

25  years  ago,  there  was  only  one  backbone.  It  was  run  by  the  Na:onal  Science   Founda:on  for  the  benefit  of  various  researchers  and  government  agencies.  Regional   networks  connected  to  the  NSF  backbone  but,  with  only  one  backbone,  there  was   only  one  source  of  addressing  and  of  ul:mate  rou:ng  decisions.   3  

As  other,  commercial  networks  grew  up,  they  interconnected  with  the  NSFNET  to   exchange  email  and  data  files.    They  also  found  other  ways  to  exchange  data  among   themselves,  but  s:ll  relied  on  the  NSFnet  as  the  ul:mate  authority  on  addressing  and   rou:ng.   4  

With  the  advent  of  the  World  Wide  Web  and  the  Mosaic  browser,  Internet  growth   accelerated  and  the  NSF  sought  a  way  to  get  out  of  the  backbone  business.    Part  of   this  required  development  of  a  new  rou:ng  protocol  (BGP,  on  which  more  later)   which  went  on  within  the  IETF  between  1991-­‐1994.  Part  of  this  required  establishing   four  Network  Access  Points  (NAPs)  where  backbone  providers  would  exchange  traffic   des:ned  for  other  backbones.   Of  course,  each  backbone  provider  had  its  own  network  that  enabled  all  connected   users  and  content  providers  to  communicate  with  one  another.  However,  users  were   not  interested  in  communica:ng  just  with  just  those  other  users  connected  to  the   same  backbone  provider.  They  wanted  to  communicate  with  any  user  and  any   content  provider,  regardless  of  backbone  provider.  To  offer  universal  connec:vity,   backbone  providers  interconnected  at  NAPs  (and  elsewhere)  to  exchange  traffic   des:ned  for  each  other’s  users.  It  is  these  interconnec:ons  that  make  the  Internet   the  “network  of  networks”  that  it  is  today.   Finally,  in  April  1995,  the  NSF  stopped  providing  backbone  services  and  the   commercial  Internet  was  born.   5  

In  order  to  provide  complete  Internet  access,  the  backbone  providers  had  to   exchange  traffic  with  each  other.    What’s  more,  the  NSFNET  backbone  had  facilitated   open  traffic  exchange  at  many  levels,  so  there  were  many  peering  agreements  at  first.   But  the  Internet  was  also  growing  rapidly,  requiring  significant  capital  investments.    At   a  minimum,  investors  wanted  to  see  a  path  to  a  return  on  their  investment.   6  

And,  with  just  six-­‐seven  full  backbones  networks  in  existence,  the  backbone  ISPs   began  to  realize  they  had  the  makings  of  a  cartel.     7  

As  a  cartel,  none  of  the  backbone  operators  had  to  provide  free  peering  to  regional,   local  or  other  smaller  networks.    Instead  they  could  sell  them  “Internet  Transit”   service  –  a  service  that  delivers  packets  to  the  rest  of  the  Internet.   Gradually  (and  some:mes  abruptly),  peering  rules  became  quite  exclusive.  To  peer   with  the  backbone,  you  had  to  be  present  at  all  major  NAPs,  you  had  to  have  a   significant  amount  of  traffic  and  that  traffic  had  to  be  roughly  symmetric.  This  had  an   immediate  impact  on  many  Tier  2  operators,  some  of  which  were  growing  more   rapidly  than  the  backbones.   8  

De-­‐peering  also  impacted  cable  companies,  several  major  content  hos:ng  networks   and  some  large  savvy  content  providers.  These  folks  realized  that,  even  if  they  had  to   buy  “Internet  Transit”  from  a  backbone  provider,  they  could  reduce  what  they  paid   the  backbone  providers  by  exchanging  traffic  among  themselves.   9  

By  2002,  donut  peering  had  emerged.  The  Tier  2  ISPs,  cable  companies  and  content   providers  had  built  a  ring  around  the  cartel,  largely  rendering  the  original  cartel   irrelevant.   10  

Indeed,  many  Tier  2  providers  now  had  interna:onal  networks  and  offered  lower   latency  &/or  be3er  pricing.    By  the  early  2000s,  the  Internet  was  substan:ally  more   distributed.   11  

The  third  wave,  which  started  in  the  early  2000s  and  is  s:ll  evolving  today,  was  the   advent  of  Content  Distribu:on  Networks  (CDNs).  CDNs  may  have  limited,  private  or   no  communica:ons  infrastructure  of  their  own,  instead  they  distribute  content   servers  in  what  is  effec:vely  an  overlay  network.    Akamai  and  Limelight  created  early   CDNs.  Today,  Google,  Amazon  and  Level  3  also  run  content  distribu:on  networks  and   Nejlix  has  begun  deploying  their  own  CDN.   12  

Typically,  major  CDNs  supply  their  servers  and  remotely  manage  them,  but  local  ISPs   install  them  and  pay  for  electricity  and  rack  space.    This  is  good  business  for  the  local   ISP  as  it  reduces  latency  for  their  customers  and  reduces  the  amount  of  upstream   Internet  transit  service  they  must  pay  for.   13  

The  past  20  years  have  seen  enormous  turbulence  among  those  providing  the  core  of   the  Internet.    The  original  backbone  networks  have  survived,  but  their  ownership  has   gone  through  a  series  of  bankruptcies,  mergers  and  acquisi:ons.    Meanwhile  the   number  of  networks  par:cipa:ng  in  the  Internet  backbone  has  grown  from  6  to  over   6000.   14  

I’ve  been  bandying  around  the  terms  “Peering”  and  “Internet  Transit.”    Let  me  explain   exactly  how  they  differ.   Internet  Transit  is  a  service  where  the  upstream  ISP  commits  to  deliver  traffic  to  any   valid  Internet  address.  It’s  typically  priced  in  $/Mbps/Month  and  the  Mbps  of  traffic  is   determined  by  measuring  traffic  levels  every  five  minutes  and  then  compu:ng  the   95th  percen:le  of  all  those  measurements  during  the  month.   Now  suppose  I’m  ISP1.  I  have  a  router  in  a  regional  data  center  where  I  buy  Internet   Transit  services,  but  I  no:ce  that  4%  of  my  traffic  is  to  my  compe:tor,  ISP2,  and  he   happens  to  have  a  router  in  the  same  regional  data  center  just  a  few  hundred  feet   away  from  mine.    He’s  my  compe:tor,  but  we  could  each  save  4%  of  our  monthly  bills   for  Internet  transit  if  we  agree  to  locally  exchange  the  traffic  that’s  des:ned  for  each   other’s  networks.   15  

No:ce  that  we’re  only  exchanging  traffic  that  originates  with  a  customer  of  one  ISP   and  terminates  with  a  customer  of  the  other  peered  ISP.   16  

ISP2  may  have  other  connec:ons  to  other  ISPs,  but  these  are  not  involved  (or  even   visible)  to  the  peering  arrangement  with  ISP1.   That’s  the  key  difference.    Peering  is  traffic  exchange  involving  only  those  addresses   that  are  served  by  the  two  peers.    Transit  involves  handling  packets  that  will  be   passed  off  to  one  or  more  addi:onal  networks.   17  

But  whether  it’s  peering  or  transit,  what  is  actually  exchanged  and  how  does  it  work?   Here  things  are  remarkably  stable.    Operators  may  exchange  other  kinds  of  traffic   (MPLS,  Carrier  Ethernet)  for  other  services,  but  for  Internet  traffic,  they  exchange  IP   packets  (mostly  IPv4)  and  they  nego:ate  routes  using  Border  Gateway  Protocol  (BGP).     IPv4  is  essen:ally  unchanged  for  over  30  years  and  the  current  version  of  BGP  has   had  only  minor  tweaks  since  it  was  deployed  20  years  ago.   Business  arrangements  have  been  turbulent,  but  the  technology  has  been  remarkably   stable.   18  

To  get  a  be3er  understanding  of  BGP,  suppose  I’m  running  BGP  on  my  edge  router   there  on  the  lel.  There  are  two  ISPs  I  wish  to  exchange  traffic  with  (either  peering  or   transit).    In  par:cular,  I’m  interested  in  gemng  traffic  to  address  blocks  A,  B  &  C.   My  router  starts  by  establishing  BGP  sessions  with  the  edge  routers  at  each  ISP.   19  

Once  the  sessions  are  up,  I  get  an  announcement  from  the  edge  router  at  ISP1  saying   it’s  prepared  to  deliver  traffic  to  address  block  A  over  a  route  that  has  three  hops  and   traffic  for  address  block  B  over  a  route  that  has  one  hop.   20  

This  is  followed  by  an  announcement  from  ISP2  saying  they  can  deliver  traffic  to   address  block  B  in  two  hops  or  to  address  block  C  in  two  hops.   Now,  I  have  to  make  some  decisions.   21  

First  these  announcements  come  from  other  organiza:ons  who  may  or  may  not  be   competent.    Should  I  believe  ISP1  when  he  says  he  can  deliver  traffic  to  address  block   B  in  just  one  hop?   A  classic  example  of  mistakes  that  can  happen  occurred  in  Feb  2008  when  the   government  of  Pakistan  told  Pakistan  Telecom  to  block  traffic  to  YouTube  because   YouTube  was  hos:ng  blasphemous  videos.  The  engineers  at  Pakistan  Telecom   complied  by  crea:ng  a  very  specific  route  for  just  the  YouTube  addresses  (part  of  a   larger  Google  address  block).  Request  packets  that  matched  this  specific  route  were   sent  to  a  “black  hole  server,”  i.e.  a  server  that  dropped  each  packet  it  received.   Unfortunately,  this  black  hole  route  leaked  out  to  the  large  interna:onal  carrier,  Hong   Kong-­‐based  PCCW.    PCCW  didn’t  have  route  filtering  in  place  on  this  par:cular  link   and  they  passed  the  black  hole  route  around  the  world.    Over  90  major  ISPs   erroneously  accepted  this  route  and  for  more  than  two  hours  YouTube  was  dark  while   almost  all  the  world’s  YouTube  requests  went  to  the  black  hole  server  in  Pakistan.   So  you  can’t  always  trust  your  neighbor,  however  competent  they  may  have  seemed   in  the  past.   There  are  many  addi:onal  considera:ons.    For  example,  certain  routes  may  have   preferen:al  pricing  up  to  a  certain  commitment  level  but  become  expensive  at  higher   traffic  levels.    So  the  choice  of  which  adver:sed  route  to  use  can  involve  some  quite   complex  considera:ons.   22  

To  give  you  a  sense  of  the  business  trade  offs  that  go  on,  I  have  two  examples.   The  first  is  a  friend  of  mine  who  formed  a  fixed  wireless  ISP  in  southeastern  Illinois  a   few  years  ago.    Because  he  was  located  in  farm  country,  the  only  way  he  could  get  an   Internet  connec:on  was  by  buying  Internet  Transit  service  (called  Direct  Internet   Access  or  DIA)  from  Ameritech  (now  AT&T)  the  local  telephone  monopoly.    His  price   was  more  than  100x  what  Internet  Transit  would  have  cost  him  in  Chicago,  but  there   were  no  compe:ng  fiber  routes  through  his  area  and  even  if  he’d  been  close  to  a  long   distance  fiber  route  (say  between  Chicago  and  St  Louis),  local  connec:ons  to  long   distance  fiber  are  extremely  expensive  or,  more  olen,  just  not  available.   Once  his  business  was  up  and  running,  my  friend  spent  many  days  driving  to  and  from   Chicago  looking  for  tall  buildings  and  talking  to  building  owners.    Eventually  he  build  a   series  of  four  wireless  links  (totaling  more  than  70  miles)  which  connected  him  to   Chicago.  In  Chicago,  he  signed  up  for  a  monthly  recurring  charge  for  rack  space,  for   roof  rights  on  the  Chicago  data  center  and  for  a  cable  from  his  rack  to  their  “meet  me   room.”    He’d  also  promised  free  high  speed  Internet  service  to  three  building  owners,   downstate,  who  gave  him  roof  access  on  the  route  to  Chicago.    But  now  that  he  was   connected  in  Chicago,  he  could  purchase  Internet  transit  from  any  of  a  dozen   compe:ng  carriers  (at  a  :ny  frac:on  of  what  he  was  paying  AT&T).    Although  he  had   spent  nearly  $100K  (and  untold  man  hours)  pumng  this  wireless  route  together,  he   figured  his  payback  was  9  weeks.    Loca:on  ma3ers!   The  second  thing  that  happened  was,  as  his  total  traffic  grew  he  began  to  qualify  for   peering  with  major  content  providers  like  Google  and  Akamai.    This  cut  further  cut  his   costs  for  Internet  transit.   23  

The  second  example  is  only  approximate,  but  representa:ve.    I  don’t  have  the  actual   numbers  on  YouTube’s  traffic  or  their  costs  during  the  20  months  between  their   founding  in  Feb  2005  and  their  purchase  by  Google  in  Oct-­‐Nov  2006,  but  I  can  tell  you   that  one  of  their  early  employees  was  a  “peering  coordinator”  who  showed  up  at   NANOG  mee:ngs  early  in  2006.    In  early  2006,  there  was  already  a  great  interest  in   peering  with  YouTube.       By  the  summer  of  2006,  YouTube  was  the  5th  most  traffic’d  website  in  the  world.    They   were  s:ll  only  peering  in  Palo  Alto,  but  anyone  with  a  router  in  Palo  Alto  was   interested  in  offloading  their  YouTube  traffic.  And  any  Tier  one  carrier  that  didn’t  peer   with  YouTube  would  quickly  find  traffic  ra:os  going  unbalanced  on  links  where  they   handed  off  YouTube  traffic  to  someone  who  was  peering  with  YouTube.   I’m  not  showing  YouTube’s  costs  going  to  zero,  but  they  clearly  did  not  increase  (and   likely  went  down)  as  YouTube’s  traffic  grew!   24  

Bill  Woodcock  and  Vijay  Adhikari  of  Packet  Clearing  House  did  a  very  comprehensive   survey  of  backbone  ISPs  in  2011  gemng  a  remarkable  86%  response  rate.    All  the   internal  indica:ons  are  this  survey  yielded  very  high  quality  data.   Several  interes:ng  things  emerged  from  this  data.    Most  notably,  many  operators   publish  a  set  of  peering  requirements,  and  these  typically  include  an  NDA.    But  if  you   meet  the  requirements,  there  are  no  formal  contracts!    These  are  handshake   agreements.   25  

One  interes:ng  thing  was,  to  the  extent  there  are  contracts  between  operators  in   different  countries,  for  example  the  NDAs,  the  choice  of  governing  law  always  favors   the  country  with  stable  ins:tu:ons,  minimum  corrup:on  and  a  func:oning  judiciary.   26  

In  terms  of  how  the  Internet  backbone  is  evolving,  the  most  interes:ng  thing  to   emerge  was  the  rise  of  mul:-­‐lateral  peering.    These  are  arrangements  that  started  in   Asia  and  selected  loca:ons  in  Europe.    We  haven’t  seen  this  in  the  US  yet,  but  there  is   an  organiza:on,  “open-­‐,”  backed  by  Google  and  Amazon  among  others,  that  is   trying  to  foster  the  spread  of  mul:-­‐lateral  peering.   27  

Mul:-­‐lateral  peering  drama:cally  reduces  the  number  of  BGP  sessions  one  must   configure  and  manage,  thus  facilita:ng  more  peering.    With  bi-­‐lateral  peering,  there   is  a  separate  BGP  session  for  every  peer.   28  

In  mul:-­‐lateral  peering,  one  organiza:on  –  perhaps  a  co-­‐op  or  a  vendor  –  provides  a   single  route  server.    Each  par:cipant  establishes  a  single  BGP  session  to  this  server.     Typically,  the  route  server  includes  session-­‐specific  configura:on  which  allows  you   some  of  the  flexibility  you  would  have  had  with  N  bi-­‐lateral  peering  sessions  but,  to   get  started,  you  can  ignore  all  that  and  just  establish  one  simple  BGP  session  that   reaches  hundreds  of  peers.   29  

This  graph  shows  the  number  of  IP  addresses  handled  by  various  carriers  as  a   func:on  of  how  many  peering  agreements  those  carriers  have.    You  can  see  that   some  of  the  original  Tier  1s  are  s:ll  visible  in  the  upper  lel,  but  otherwise,  the   Internet  backbone  is  very  distributed.    And  this  graph  is  based  on  addresses  handled,   not  on  traffic  carried.   30  

When  we  look  at  traffic,  the  top  ISPs  are  quite  distributed.    Also,  we  can  see  what   happens  when  the  large  carrier  (Level  3  at  the  top)  buys  the  second  largest  carrier   (Global  Crossing  in  grey  just  below)  as  happened  in  April  2012.    Both  networks   immediately  saw  a  drop  in  traffic  as  customers  who  wanted  redundant  connec:ons   dropped  one  of  their  connec:ons  to  the  now  merged  business.    Then  over  :me,  both   networks  see  further  drops  in  traffic  as  the  rest  of  the  players  rearrange  their   networks.    Also,  note  that  this  traffic  diagram  only  deals  with  ISPs  that  offer  Internet   transit  services.    The  second  largest  network  in  the  world,  by  traffic,  is  Google.  If   Google  were  shown  on  this  graph,  it  would  appear  between  the  black  and  grey  lines.   So  the  Internet  is  very  distributed  and,  as  Rensys  notes  in  their  report,  the  rela:ve   market  share  of  the  backbone  carriers  as  a  group  has  been  falling  over  the  past   decade.     In  ~20    years  of  the  commercial  Internet,  no  one  has  been  able   to  gain  control  of  the  Internet  backbone.  In  the  1990s,  the  original  gang  of  ~six   backbone  providers  thought  they  had  an  oligopoly  (a  cartel),  but  by  2002,  second   :er  backbones  used  "donut    peering”  to  eliminate  the  original  :er  ones’   leverage.  Since  2000,  we've  seen  the  emergence  of  mul:ple  CDNs  (Akamai,  Level  3,   Google,  Limelight,  plus  Amazon,  Nejlix,  and  others  in  the  making)  which  have  further   diluted  any  a3empt  to  monopolize  the  backbone.    Also  over  the  past  20+  years,  we've   seen  an  explosion  in  the  number  of  buildings  where  some  kind  of  peering  takes  place.   In  short,  no  one  has  been  able  to  monopolize  the  Internet  backbone.    Now  we’re   seeing  the  emergence  of  mul:-­‐lateral  peering  and  even  more  backbone  par:cipants.   31  

The  Internet  backbone  is  a  very  interes:ng  phenomenon.    It’s  essen:ally  unregulated.   IANA  (the  body  that  supervises  the  assignment  of  addresses  and  other  protocol   number  assignments)  provides  only  coordina:on.  If  IANA  withheld  or  manipulated   assignments,  their  func:on  could  be  quickly  and  informally  bypassed.   Recently  we’ve  heard  a  lot  about  “regula:ng  the  Internet”  especially  since  the   revela:ons  of  NSA  spying.  But  most  such  discussion  is  happening  without  any   understanding  of  how  the  Internet  backbone  actually  works.   Forecasts  are  iffy,  but  the  current  system  is  extremely  successful  and  extremely   robust,  so  I  am  op:mis:c  the  Internet  will  con:nue  to  grow,  indefinitely.   32  


Add a comment

Related presentations

Related pages

Internet peering, with annotations - Technology

The Internet backbone consists of just over 6000 independent networks that exchange traffic in fashions that are not well understood outside of the ...
Read more

Internet peering, graphics only - Technology -

These are the graphics (in higher resolution) for my presentation, Internet Peering with annotations. See "Internet Peering, with annotations" for details.
Read more

Mapping Interconnection in the Internet: Colocation ...

The proposal for "Mapping Interconnection in the Internet: Colocation, Connectivity and Congestion ... annotations , which allows us ... internet peering ...
Read more

CAIDA: Center for Applied Internet Data Analysis

AS Relationships with geographic annotations: ... The Remote Peering Jedi: Nov 11: NANOG68: ... "CAIDA adds geo-tags to its Internet AS dataset" (The Register)
Read more

snmp xml free download - SourceForge

It allows you to expose your spring beans via SNMP transparently using XML or annotations. ... Internet BGP Peering ... visualize BGP Peering topology by ...
Read more

The Internet is flat - Association for Computing Machinery

The Internet is flat: ... Kc Claffy, Measuring the evolution of internet peering agreements, ... images, text annotations, ...
Read more

Program annotations -

Program annotations: feedback ... This paper describes elements in the development of a board game for teaching Internet peering as part of an Internet ...
Read more

Orbis: Rescaling Degree Correlations to Generate Annotated ...

Orbis: Rescaling Degree Correlations to Generate Annotated Internet Topologies Priya Mahadevan UC San Diego Calvin Hubble UC San Diego
Read more