Published on March 14, 2014
Why care? Peering and Transit are li3le understood and yet this is the heart of Internet infrastructure economics and it is the growth of Internet infrastructure that ul:mately funds most of us here at OFC. I going to describe how thousands of independent organiza:ons compete but also exchange traﬃc and con:nually grow an ever more distributed Internet infrastructure. 1
There are more than 3 billion Internet users and, with the advent of very low cost Android smart phones ($24 in India in Feb 2014), it’s likely that number will double in just a few years. There are also tens of millions of local networks (perhaps more than 100 million based just on the number of WiFi routers that have been sold). They connect to the Internet through tens of thousands of ISPs which may be classiﬁed as Access, Aggrega:on or Backbone, or as Local, Regional, Na:onal or Interna:onal but, of course, many ISPs cross these boundaries. For today, I’m going to focus on the 6000 or so major ISPs that form today’s Internet backbone. But to understand today’s complex environment, it’s useful to see how it emerged. 2
25 years ago, there was only one backbone. It was run by the Na:onal Science Founda:on for the beneﬁt of various researchers and government agencies. Regional networks connected to the NSF backbone but, with only one backbone, there was only one source of addressing and of ul:mate rou:ng decisions. 3
As other, commercial networks grew up, they interconnected with the NSFNET to exchange email and data ﬁles. They also found other ways to exchange data among themselves, but s:ll relied on the NSFnet as the ul:mate authority on addressing and rou:ng. 4
With the advent of the World Wide Web and the Mosaic browser, Internet growth accelerated and the NSF sought a way to get out of the backbone business. Part of this required development of a new rou:ng protocol (BGP, on which more later) which went on within the IETF between 1991-‐1994. Part of this required establishing four Network Access Points (NAPs) where backbone providers would exchange traﬃc des:ned for other backbones. Of course, each backbone provider had its own network that enabled all connected users and content providers to communicate with one another. However, users were not interested in communica:ng just with just those other users connected to the same backbone provider. They wanted to communicate with any user and any content provider, regardless of backbone provider. To oﬀer universal connec:vity, backbone providers interconnected at NAPs (and elsewhere) to exchange traﬃc des:ned for each other’s users. It is these interconnec:ons that make the Internet the “network of networks” that it is today. Finally, in April 1995, the NSF stopped providing backbone services and the commercial Internet was born. 5
In order to provide complete Internet access, the backbone providers had to exchange traﬃc with each other. What’s more, the NSFNET backbone had facilitated open traﬃc exchange at many levels, so there were many peering agreements at ﬁrst. But the Internet was also growing rapidly, requiring signiﬁcant capital investments. At a minimum, investors wanted to see a path to a return on their investment. 6
And, with just six-‐seven full backbones networks in existence, the backbone ISPs began to realize they had the makings of a cartel. 7
As a cartel, none of the backbone operators had to provide free peering to regional, local or other smaller networks. Instead they could sell them “Internet Transit” service – a service that delivers packets to the rest of the Internet. Gradually (and some:mes abruptly), peering rules became quite exclusive. To peer with the backbone, you had to be present at all major NAPs, you had to have a signiﬁcant amount of traﬃc and that traﬃc had to be roughly symmetric. This had an immediate impact on many Tier 2 operators, some of which were growing more rapidly than the backbones. 8
De-‐peering also impacted cable companies, several major content hos:ng networks and some large savvy content providers. These folks realized that, even if they had to buy “Internet Transit” from a backbone provider, they could reduce what they paid the backbone providers by exchanging traﬃc among themselves. 9
By 2002, donut peering had emerged. The Tier 2 ISPs, cable companies and content providers had built a ring around the cartel, largely rendering the original cartel irrelevant. 10
Indeed, many Tier 2 providers now had interna:onal networks and oﬀered lower latency &/or be3er pricing. By the early 2000s, the Internet was substan:ally more distributed. 11
The third wave, which started in the early 2000s and is s:ll evolving today, was the advent of Content Distribu:on Networks (CDNs). CDNs may have limited, private or no communica:ons infrastructure of their own, instead they distribute content servers in what is eﬀec:vely an overlay network. Akamai and Limelight created early CDNs. Today, Google, Amazon and Level 3 also run content distribu:on networks and Nejlix has begun deploying their own CDN. 12
Typically, major CDNs supply their servers and remotely manage them, but local ISPs install them and pay for electricity and rack space. This is good business for the local ISP as it reduces latency for their customers and reduces the amount of upstream Internet transit service they must pay for. 13
The past 20 years have seen enormous turbulence among those providing the core of the Internet. The original backbone networks have survived, but their ownership has gone through a series of bankruptcies, mergers and acquisi:ons. Meanwhile the number of networks par:cipa:ng in the Internet backbone has grown from 6 to over 6000. 14
I’ve been bandying around the terms “Peering” and “Internet Transit.” Let me explain exactly how they diﬀer. Internet Transit is a service where the upstream ISP commits to deliver traﬃc to any valid Internet address. It’s typically priced in $/Mbps/Month and the Mbps of traﬃc is determined by measuring traﬃc levels every ﬁve minutes and then compu:ng the 95th percen:le of all those measurements during the month. Now suppose I’m ISP1. I have a router in a regional data center where I buy Internet Transit services, but I no:ce that 4% of my traﬃc is to my compe:tor, ISP2, and he happens to have a router in the same regional data center just a few hundred feet away from mine. He’s my compe:tor, but we could each save 4% of our monthly bills for Internet transit if we agree to locally exchange the traﬃc that’s des:ned for each other’s networks. 15
No:ce that we’re only exchanging traﬃc that originates with a customer of one ISP and terminates with a customer of the other peered ISP. 16
ISP2 may have other connec:ons to other ISPs, but these are not involved (or even visible) to the peering arrangement with ISP1. That’s the key diﬀerence. Peering is traﬃc exchange involving only those addresses that are served by the two peers. Transit involves handling packets that will be passed oﬀ to one or more addi:onal networks. 17
But whether it’s peering or transit, what is actually exchanged and how does it work? Here things are remarkably stable. Operators may exchange other kinds of traﬃc (MPLS, Carrier Ethernet) for other services, but for Internet traﬃc, they exchange IP packets (mostly IPv4) and they nego:ate routes using Border Gateway Protocol (BGP). IPv4 is essen:ally unchanged for over 30 years and the current version of BGP has had only minor tweaks since it was deployed 20 years ago. Business arrangements have been turbulent, but the technology has been remarkably stable. 18
To get a be3er understanding of BGP, suppose I’m running BGP on my edge router there on the lel. There are two ISPs I wish to exchange traﬃc with (either peering or transit). In par:cular, I’m interested in gemng traﬃc to address blocks A, B & C. My router starts by establishing BGP sessions with the edge routers at each ISP. 19
Once the sessions are up, I get an announcement from the edge router at ISP1 saying it’s prepared to deliver traﬃc to address block A over a route that has three hops and traﬃc for address block B over a route that has one hop. 20
This is followed by an announcement from ISP2 saying they can deliver traﬃc to address block B in two hops or to address block C in two hops. Now, I have to make some decisions. 21
First these announcements come from other organiza:ons who may or may not be competent. Should I believe ISP1 when he says he can deliver traﬃc to address block B in just one hop? A classic example of mistakes that can happen occurred in Feb 2008 when the government of Pakistan told Pakistan Telecom to block traﬃc to YouTube because YouTube was hos:ng blasphemous videos. The engineers at Pakistan Telecom complied by crea:ng a very speciﬁc route for just the YouTube addresses (part of a larger Google address block). Request packets that matched this speciﬁc route were sent to a “black hole server,” i.e. a server that dropped each packet it received. Unfortunately, this black hole route leaked out to the large interna:onal carrier, Hong Kong-‐based PCCW. PCCW didn’t have route ﬁltering in place on this par:cular link and they passed the black hole route around the world. Over 90 major ISPs erroneously accepted this route and for more than two hours YouTube was dark while almost all the world’s YouTube requests went to the black hole server in Pakistan. So you can’t always trust your neighbor, however competent they may have seemed in the past. There are many addi:onal considera:ons. For example, certain routes may have preferen:al pricing up to a certain commitment level but become expensive at higher traﬃc levels. So the choice of which adver:sed route to use can involve some quite complex considera:ons. 22
To give you a sense of the business trade oﬀs that go on, I have two examples. The ﬁrst is a friend of mine who formed a ﬁxed wireless ISP in southeastern Illinois a few years ago. Because he was located in farm country, the only way he could get an Internet connec:on was by buying Internet Transit service (called Direct Internet Access or DIA) from Ameritech (now AT&T) the local telephone monopoly. His price was more than 100x what Internet Transit would have cost him in Chicago, but there were no compe:ng ﬁber routes through his area and even if he’d been close to a long distance ﬁber route (say between Chicago and St Louis), local connec:ons to long distance ﬁber are extremely expensive or, more olen, just not available. Once his business was up and running, my friend spent many days driving to and from Chicago looking for tall buildings and talking to building owners. Eventually he build a series of four wireless links (totaling more than 70 miles) which connected him to Chicago. In Chicago, he signed up for a monthly recurring charge for rack space, for roof rights on the Chicago data center and for a cable from his rack to their “meet me room.” He’d also promised free high speed Internet service to three building owners, downstate, who gave him roof access on the route to Chicago. But now that he was connected in Chicago, he could purchase Internet transit from any of a dozen compe:ng carriers (at a :ny frac:on of what he was paying AT&T). Although he had spent nearly $100K (and untold man hours) pumng this wireless route together, he ﬁgured his payback was 9 weeks. Loca:on ma3ers! The second thing that happened was, as his total traﬃc grew he began to qualify for peering with major content providers like Google and Akamai. This cut further cut his costs for Internet transit. 23
The second example is only approximate, but representa:ve. I don’t have the actual numbers on YouTube’s traﬃc or their costs during the 20 months between their founding in Feb 2005 and their purchase by Google in Oct-‐Nov 2006, but I can tell you that one of their early employees was a “peering coordinator” who showed up at NANOG mee:ngs early in 2006. In early 2006, there was already a great interest in peering with YouTube. By the summer of 2006, YouTube was the 5th most traﬃc’d website in the world. They were s:ll only peering in Palo Alto, but anyone with a router in Palo Alto was interested in oﬄoading their YouTube traﬃc. And any Tier one carrier that didn’t peer with YouTube would quickly ﬁnd traﬃc ra:os going unbalanced on links where they handed oﬀ YouTube traﬃc to someone who was peering with YouTube. I’m not showing YouTube’s costs going to zero, but they clearly did not increase (and likely went down) as YouTube’s traﬃc grew! 24
Bill Woodcock and Vijay Adhikari of Packet Clearing House did a very comprehensive survey of backbone ISPs in 2011 gemng a remarkable 86% response rate. All the internal indica:ons are this survey yielded very high quality data. Several interes:ng things emerged from this data. Most notably, many operators publish a set of peering requirements, and these typically include an NDA. But if you meet the requirements, there are no formal contracts! These are handshake agreements. 25
One interes:ng thing was, to the extent there are contracts between operators in diﬀerent countries, for example the NDAs, the choice of governing law always favors the country with stable ins:tu:ons, minimum corrup:on and a func:oning judiciary. 26
In terms of how the Internet backbone is evolving, the most interes:ng thing to emerge was the rise of mul:-‐lateral peering. These are arrangements that started in Asia and selected loca:ons in Europe. We haven’t seen this in the US yet, but there is an organiza:on, “open-‐ix.org,” backed by Google and Amazon among others, that is trying to foster the spread of mul:-‐lateral peering. 27
Mul:-‐lateral peering drama:cally reduces the number of BGP sessions one must conﬁgure and manage, thus facilita:ng more peering. With bi-‐lateral peering, there is a separate BGP session for every peer. 28
In mul:-‐lateral peering, one organiza:on – perhaps a co-‐op or a vendor – provides a single route server. Each par:cipant establishes a single BGP session to this server. Typically, the route server includes session-‐speciﬁc conﬁgura:on which allows you some of the ﬂexibility you would have had with N bi-‐lateral peering sessions but, to get started, you can ignore all that and just establish one simple BGP session that reaches hundreds of peers. 29
This graph shows the number of IP addresses handled by various carriers as a func:on of how many peering agreements those carriers have. You can see that some of the original Tier 1s are s:ll visible in the upper lel, but otherwise, the Internet backbone is very distributed. And this graph is based on addresses handled, not on traﬃc carried. 30
When we look at traﬃc, the top ISPs are quite distributed. Also, we can see what happens when the large carrier (Level 3 at the top) buys the second largest carrier (Global Crossing in grey just below) as happened in April 2012. Both networks immediately saw a drop in traﬃc as customers who wanted redundant connec:ons dropped one of their connec:ons to the now merged business. Then over :me, both networks see further drops in traﬃc as the rest of the players rearrange their networks. Also, note that this traﬃc diagram only deals with ISPs that oﬀer Internet transit services. The second largest network in the world, by traﬃc, is Google. If Google were shown on this graph, it would appear between the black and grey lines. So the Internet is very distributed and, as Rensys notes in their report, the rela:ve market share of the backbone carriers as a group has been falling over the past decade. In ~20 years of the commercial Internet, no one has been able to gain control of the Internet backbone. In the 1990s, the original gang of ~six backbone providers thought they had an oligopoly (a cartel), but by 2002, second :er backbones used "donut peering” to eliminate the original :er ones’ leverage. Since 2000, we've seen the emergence of mul:ple CDNs (Akamai, Level 3, Google, Limelight, plus Amazon, Nejlix, and others in the making) which have further diluted any a3empt to monopolize the backbone. Also over the past 20+ years, we've seen an explosion in the number of buildings where some kind of peering takes place. In short, no one has been able to monopolize the Internet backbone. Now we’re seeing the emergence of mul:-‐lateral peering and even more backbone par:cipants. 31
The Internet backbone is a very interes:ng phenomenon. It’s essen:ally unregulated. IANA (the body that supervises the assignment of addresses and other protocol number assignments) provides only coordina:on. If IANA withheld or manipulated assignments, their func:on could be quickly and informally bypassed. Recently we’ve heard a lot about “regula:ng the Internet” especially since the revela:ons of NSA spying. But most such discussion is happening without any understanding of how the Internet backbone actually works. Forecasts are iﬀy, but the current system is extremely successful and extremely robust, so I am op:mis:c the Internet will con:nue to grow, indeﬁnitely. 32
The Internet backbone consists of just over 6000 independent networks that exchange traffic in fashions that are not well understood outside of the ...
These are the graphics (in higher resolution) for my presentation, Internet Peering with annotations. See "Internet Peering, with annotations" for details.
The proposal for "Mapping Interconnection in the Internet: Colocation, Connectivity and Congestion ... annotations , which allows us ... internet peering ...
AS Relationships with geographic annotations: ... The Remote Peering Jedi: Nov 11: NANOG68: ... "CAIDA adds geo-tags to its Internet AS dataset" (The Register)
It allows you to expose your spring beans via SNMP transparently using XML or annotations. ... Internet BGP Peering ... visualize BGP Peering topology by ...
The Internet is flat: ... Kc Claffy, Measuring the evolution of internet peering agreements, ... images, text annotations, ...
Program annotations: feedback ... This paper describes elements in the development of a board game for teaching Internet peering as part of an Internet ...
Orbis: Rescaling Degree Correlations to Generate Annotated Internet Topologies Priya Mahadevan UC San Diego firstname.lastname@example.org Calvin Hubble UC San Diego