Inter-AS Recommendations and Traversing Multiple Provider Trust Model Issues
Standard RFC 2547 MPLS VPN networks have a simple interface to the outside of the network: IP. Any VPN that connects to the MPLS core can only send IP datagrams into the core. Internally, MPLS core networks work mostly with LSPs, using labeled packets for forwarding. Because the inside uses different packet types from the outside and because the inside format of labeled packets is not accepted on ingress, there is no way to attack an MPLS core from the outside. However, because an RFC 2547 network uses iBGP to distribute VPN routing, this mechanism cannot be used across autonomous system (AS) boundaries.
RFC 2547bis introduced the concept of Inter-AS to connect various autonomous systems together with ASBRs and to provide VPNs over all ASs involved. Figure 4-22 shows the basic concept of Inter-AS topologies, here with two autonomous systems. There is no limit on the number of ASs involved in such a setup.
Figure 4-22. A Generic Inter-AS Scenario
The key difference between this scenario and the traditional RFC 2547 networks is that, on the Inter-AS boundary shown in Figure 4-22, labeled packets may be exchanged. This changes the security exposure of an MPLS core significantly: because labeled packets are now accepted, those need to be checked to avoid attacks against the inside of the MPLS core.
Because the threat comes from exchanging labeled packets and because VPN customers can send only IP packets and not labeled packets, the threat we have to examine for this section is attacks from a connected service provider network, not attacks from a VPN customer. All the attacks described in this section assume that one of the providers in an Inter-AS scenario accidentally or deliberately makes certain configuration mistakes.
NOTE
If an Inter-AS scenario is used to connect autonomous systems that are under the operational control of a single service provider, this section can be ignored. The same applies if the autonomous systems are maintained by providers that trust each other fully.
Before discussing the various scenarios, it should be clear what we are trying to achieve by securing the Inter-AS boundaries. The goals of securing the Inter-AS boundaries are as follows:
Each SP must be able to send packets across the Inter-AS boundary into all Inter-AS VPNs (VPNs that are shared between the autonomous systems).
No SP must be able to send packets across the Inter-AS boundary into private VPNs (VPNs that are not shared between the autonomous systems).
The various autonomous systems must be secure against each other: attacking one core from another must not be possible. To achieve this, it is highly desirable to limit visibility and reachability across AS boundaries.
The goals of Inter-AS scenarios are not the following:
To provide any form of access control on the Inter-AS boundary: a given Inter-AS VPN is shared completely, without restrictions between the ASs. Specifically, there is no firewalling, packet filtering, or any other content inspection between the ASs for a given Inter-AS VPN.
To separate Inter-AS VPNs: a service provider always has full access to all Inter-AS VPNs (and none to private VPNs). It can insert any type of traffic into any of the Inter-AS VPNs; specifically, it can "misconnect" two VPNs and send traffic from Inter-AS A into Inter-AS B. (However, it also has this capability in a standard RFC 2547 network.)
As a consequence of these considerations, VPN customers must always trust all of the service providers involved in provisioning their VPN. Depending on the interconnection model chosen, service providers must trust each other or not. (Refer to Chapter 2, "A Threat Model for MPLS VPNs.") In the rest of this section, we discuss the security properties of the three interconnection models for Inter-AS.
NOTE
For this section, a thorough understanding of Chapter 10 of RFC 2547bis is required. The architectures described there will be repeated in this book only briefly as a reference. Please refer to the RFC for detailed descriptions.
Case A: VRF-to-VRF Connection on ASBRs
In case A of the Inter-AS architecture, the two autonomous systems are connected separately per Inter-AS VPN. The ASBR in this model is equivalent to a PE router with VRFs for all Inter-AS VPNs and interfaces or subinterfaces to the other AS. Figure 4-23 depicts this scenario.
Figure 4-23. Inter-AS Case A
The only interconnections are VRF based. To keep the traffic separate, each Inter-AS VPN requires a separate logical connection between the autonomous systems, and this can be achieved by using separate cables or subinterfaces, such as VLANs on an Ethernet connection.
This model is equivalent in security to a standard RFC 2547 network because, here also, all external interfaces into the core are "IP-only" interfaces. The interfaces connect directly into the VRFs of the corresponding VPNs. Specifically, there is no connection between the global routing tables of the ASBRs.
The only potential security issues in model A are
Accidental misconnection at the ASBR? When provisioning a VPN connection, service provider 1 will tell service provider 2 that VPN A is configured on a given interface or subinterface. Either service provider may make a mistake in connecting this interface into the wrong VRF, thereby connecting two VPNs that should not be connected. This risk, however, exists in all VPN environments, such as in ATM or Frame Relay, for example. It must be controlled operationally by making appropriate checks after a connection has been established.
Routing issues? The VRFs on both ASBRs will exchange routing for a given Inter-AS VPN. The usual routing security mechanisms (for example, MD5 security) must be deployed here. Most importantly, the number of prefixes must be limited on each ASBR within a VRF to avoid the ASBR running out of memory because a single VPN announces too many routes. This could happen, for example, if the whole Internet routing table accidentally gets propagated into the VPN. See Chapter 5, "Security Recommendations," for more details. Note that these precautions also need to be taken on each PE. Again, the ASBR in this model is equivalent to a PE.
For the service provider, the security exposure is exactly the same as in a single-AS MPLS network: its network cannot be attacked from the other service provider if it is appropriately secured. See Chapter 3, "MPLS Security Analysis," for a detailed discussion of the security in this environment.
For the VPN customer, the security risk is also equivalent to that of a single-AS MPLS network, except that in the Inter-AS model the customer has to trust all the service providers involved in the provisioning of the VPN: any of the providers could make the VPN insecure?when misconfiguring routers, for example.
This model has a number of other advantages as well: because each VPN is connected individually, QoS control for a given VPN is easy. Also, because the ASBR is equivalentto a PE in this model, existing provisioning systems can be used for the provisioning of the Inter-AS connection. Essentially, an ASBR in this model thinks it is connecting to a CE for each VRF; the fact that there is another MPLS core "on the other side" is not visible.
However, model A is not very scalable: each Inter-AS VPN must be configured with a VFR and an interface or subinterface on the ASBR. This requires memory for all the VPN routes, plus potentially a large range of subinterfaces. To give an example, the limit on a Cisco 12000 router with POS frame encapsulation is in the range of 200 VPNs. Models B and C were, therefore, designed to improve the scalability of the Inter-AS architecture. However, the improved scalability comes at the cost of higher security exposure.
Case B: eBGP Redistribution of Labeled VPN-IPv4 Routes
As in case A, here there is only one connection between the ASs: The ASBRs are connected back to back, but this time in the global routing table, not in a VRF. The VPN-IPv4 prefixes of the Inter-AS VPNs are held in the BGP table, not in VRFs. There are also no separate interfaces per VPN in this model. Figure 4-24 depicts this architecture.
Figure 4-24. Inter-AS Case B
Because there is only one connection between the ASBRs, data plane traffic for different VPNs must be kept separate. This is done by labeling packets before sending them to the other ASBR.
NOTE
The fact that an external interface to an AS accepts labeled packets instead of just IP packets changes the security exposure completely because labeled packets are also used internally in the cores: the strict separation between "outside" and "inside" is broken!
To define which label to use for which VPN prefix, the ASBRs exchange on the control plane VPN-IPv4 routing information with labels, through eBGP. (MP-BGP, the multiprotocol extensions to BGP, are required to be able to exchange routes of the VPN-IPv4 family.)
In the following subsections, we examine the security of the control plane and data plane separately.
Control Plane
The control plane can be secured well with standard routing security measures:
Peer authentication with MD5? This feature ensures that the routing updates are coming from the correct peer.
neighbor 192.0.2.1 password si%e_3kee$i5
Route flap damping (RR-RR and ASBR-ASBR)? To prevent quick announce and withdraw messages of a prefix from increasing the CPU load, new updates are ignored after a few flaps for a certain time.
bgp dampening
Generalized TTL Security Mechanism (GTSM)? This checks the TTL of an incoming BGP update; if the TTL is lower than an expected value (normally 254), then the packet has been originated "further away" than expected and is likely bogus. GTSM is described in RFC 3682.
neighbor 192.0.2.1 ttl-security hops 1
Prefix filtering? No IP prefixes should be exchanged in this model (only VPN-IPv4); therefore, all IP prefixes should be blocked.
ip prefix-list abc deny 0.0.0.0/0 le 32
Route-target filtering? Each VPN-IPv4 route comes with a set of RTs, defining to which VPN the prefix belongs. Wrongly propagated RTs might lead to intrusions into private VPNs of the other AS.
ip extcommunity-list 101 permit RT:100:3+
By default, a PE only accepts a VPN-IPv4 prefix if it is imported into a local VRF. Because model B does not have VRFs on the PE, by default no route would be accepted. To switch off this behavior, use the command
no bgp default route-target filter
No IP routing? By default, BGP carries the IP address family. This should be switched off.
no bgp default ipv4-unicast
Measures against prefix flooding? To avoid memory exhaustion on an ASBR or RR, a maximum number of prefixes for each BGP peer should be configured.
neighbor 192.0.2.2 maximum-prefix 100 80
(Case B does not have VRFs on the ASBR; therefore, the prefix limits per VRF are not applicable here.)
Standard router security? Use standard router security features to protect the router against intrusions and DoS attacks. Receive ACLs (rACLs) or Control Plane Policing (CoPP) are important examples.
Please refer to Chapter 5 for detailed configuration examples.
NOTE
Recommendation
Configure all these security mechanisms as tightly as possible. A hacker only needs one weakness to succeed!
Following the assumptions at the beginning of this section, prefix filtering within one VPN is not required because the VPNs are shared between ASs in full. At the time of writing this book, it is not possible in IOS to filter on VPNv4 addresses. This is addressed in enhancement request CSCec32038.
VPN-IPv4 addresses contain route distinguishers (RDs). Those RDs must be chosen such that they are unique across all ASs. The convention is to use the AS number as part of the RD, which provides uniqueness. However, this is not enforced on the routers.
What can happen if the RDs are not unique across the autonomous systems? Figure 4-25 shows the flow of routing in such a case.
Figure 4-25. Consequence of RD Overlap
In the example shown in Figure 4-25, the left PE learns from a connected VPN the prefix 192.1.1/24. It is configured with an RD of 1:100 for the corresponding VRF, and therefore announces this VPN-IPv4 prefix to the route reflector (RR). The ASBR is learning a prefix from AS 2, which has the RD wrongly set to 1. This prefix happens to be the same as on AS 1. The RR will receive, therefore, two identical VPN-IPv4 routes with different next hops.
An important implementation detail: whereas PE routers can support multiple prefixes, RRs currently cannot. (This is addressed in enhancement requests CSCdu51714 and CSCdu51721.) Therefore, the RR will choose the better prefix by some metric (see "BGP Best Path Selection Process") and announce only that. The effect is that one of the paths for the VPN-IPv4 prefixes will be lost on the RR, such that one VPN might experience connectivity issues to the prefix in question. The separation of VPNs is not affected by this.
Which of the two VPN-IPv4 prefix paths will be chosen depends on the BGP decision process, which can be controlled on the corresponding router?here, the RR. The following list shows the BGP decision process. One of those criteria can be chosen to differentiate external from internal VPN-IPv4 prefixes and to give the internal one preference. For example, AS path length can be an option.
BGP Best Path Selection Process
|
Therefore, while the ASBR can be configured to keep multiple AS paths for a single VPN-IPv4 prefix in the BGP table, a decision must be taken on the RR according to this list. To avoid a potential RD overlap, we recommend using, for example, AS path length to prefer the internal path on the RR.
NOTE
Recommendation
Uniqueness of RDs across all autonomous systems is essential and must be enforced through operational means. The easiest way is for all service providers involved to follow the conventions to use the AS number as part of the RD.
Overall, the control plane can be well secured in case B. There are commands to provide security against most parameters that could be faked. One issue is that it is currently impossible to filter on VPN-IPv4 prefixes or RDs on the ASBR border. This allows the introduction of bogus prefixes within one VPN. This is probably not a serious issue in real implementations, but it must be considered.
Data Plane
To secure the data plane, two types of traffic need to be considered: IP traffic and labeled traffic. The traffic on the Inter-AS interface contains the following:
BGP traffic (TCP with destination port 179 can be originated from either side) for the control plane. Usually the BGP session is established with the directly connected interface IP addresses, not with loopbacks.
Labeled traffic (non-IP) for the data plane.
Following basic security rules, anything that should not happen should be blocked. Therefore, we strongly recommend filtering on this interface as tightly as possible. Because data traffic is exchanged with labels, most IP traffic can be blocked.
NOTE
Recommendation
On the ASBR-ASBR interface, filter all inbound IP traffic (and possibly outbound traffic, as well, to check your own policy) with the exception of BGP. Keep in mind that a BGP session may be originated from both sides, so the ACL needs to take both directions into consideration. Be conservative with exceptions to this rule!
Apart from BGP, all IP traffic is now blocked. BGP is secured with the mechanisms described in the preceding section, "Control Plane." If all these mechanisms are implemented well, IP traffic cannot create a security hole.
Therefore, the last type of traffic to consider is labeled traffic. All VPN traffic is exchanged with labels. From a security point of view, the question is whether the label or the packet underneath can be changed to create a security hole.
Labels are exchanged by the BGP peering between the ASBRs. This way, each ASBR learns from the other which label to use for which VPN-IPv4 prefix.
Therefore, the first check when looking at a labeled packet would be whether the label observed on the data plane has been assigned previously on the control plane. Figure 4-26 shows this principle.
Figure 4-26. Assignment and Use of Labels in Model B
While, therefore, the ASBR theoretically has the capability to check incoming labels, this is not supported everywhere at the time of writing this book.
Currently, an incoming label can be checked when the interface terminates in a VRF; but it cannot be checked when the interface terminates in the global routing table. Therefore, this security check currently cannot be done in model B (nor in model C). Once this feature is available, it is strongly recommended to switch it on.
In addition, if the front label is faked to match a VPN label on the other ASBR, the packet would be forwarded to this VPN, effectively providing unidirectional traffic into a nonshared VPN. Also, if the top label matches a PE label on the ASBR, the packet would be forwarded to that PE, the top label would be popped before the egress PE, and the remaining packet would be interpreted there. This would effectively also allow intrusions into nonshared VPNs, as well as into the IGP of the other AS if an IP packet is underneath the top label. Note that all of those potential attacks can only be carried out from within the service provider's core, not from connected customers.
To overcome the issue of not being able to check the incoming label, we recommend you check a specific design option where the incoming ASBR does not have any label except VPN labels of Inter-AS VPNs. This can be achieved by putting another router in front of the existing ASBR, which only knows labels to Inter-AS VPNs. Figure 4-27 shows this design. The new router is labeled "ASBR (external)" in this figure. It maintains an eBGP session to the internal ASBR, representing essentially another Inter-AS case B interconnection.
Figure 4-27. Adding a Router for Label Checking
The external ASBR holds only labels of Inter-AS VPNs in its label forwarding information base (LFIB). If a packet is received with another label that is unknown, it is dropped. The trick is that the external ASBR does not have LSPs into the core of AS 2.
Between the internal and external ASBR, another instance of model B can be used. This way, the external ASBR does not hold labels in its LFIB that correspond to LSPs to egress PEs. However, this currently requires a separate AS number to be used on the external ASBR. This can, however, be a private AS number if that doesn't clash with the other provider.
In summary, model B can be secured adequately on the control plane. On the data plane there is as of today a lacking feature in that the incoming label cannot be checked. This creates a security hole that allows traffic to be sent from the service provider into other VPNs or the core of the other service provider. To work around this, another ASBR can be added to front-end the existing ASBR. With this setup, model B can be secured adequately.
In model B, the ASBR still needs to keep the full routing information of all Inter-AS VPNs in the BGP table. This can be a scalability limitation. Model C removes this requirement as well, and makes the ASBR even more scalable.
Case C: Multi-Hop eBGP Distribution of Labeled VPN-IPv4 Routes with eBGP Redistribution of IP4 Routes
Most MPLS core networks use route reflectors (RRs) internally to scale their network. The idea in model C is to exchange the VPN-IPv4 routes directly between the RRs so that the ASBRs do not need to hold any VPN information. The ASBRs exchange routing information about the PEs and the labels needed to get to them. This way, an end-to-end LSP can be established across the autonomous systems. Figure 4-28 depicts this setup.
Figure 4-28. Inter-AS Case C
On the control plane, VPN information is exclusively exchanged between RRs. The ASBRs exchange routing and label information between the ASs involved via eBGP. On the data plane there are labeled packets for the VPN traffic, but these packets hold at least two labels: a PE label, which leads to the egress PE, and a VPN label, which identifies the VPN on that PE.
Also in model C, the control plane can be secured well with standard routing security measures. However, they now need to be applied twice: RR to RR and ASBR to ASBR. The full list is
Peer authentication with MD5 (RR-RR and ASBR-ASBR)? This feature ensures that the routing updates are coming from the correct peer.
neighbor 192.0.2.1 password si%e_3kee$i5
Route flap damping (RR-RR and ASBR-ASBR)? To prevent quick announce and withdraw messages of a prefix from increasing the CPU load, new updates are ignored after a few flaps for a certain time.
bgp dampening
Generalized TTL Security Mechanism (GTSM) (RR-RR and ASBR-ASBR)? This checks the TTL of an incoming BGP update; if the TTL is lower than an expected value (normally 254), then the packet has been originated "further away" than expected and is likely bogus. GTSM is described in RFC 3682.
neighbor 192.0.2.1 ttl-security hops 1
The ASBRs are usually only one hop away, which makes the configuration of GTSM easy. For the RR-RR peering, this is more complex because potential backup paths need to be considered as well. The maximum TTL value to be configured is the value of the longest possible path between the RRs.
Prefix filtering (RR-RR)? No IP prefixes are exchanged between RRs (only VPN-IPv4); therefore, all IP prefixes should be blocked.
ip prefix-list abc deny 0.0.0.0/0 le 32
As mentioned in the section for model B, VPN-IPv4 routes cannot currently be filtered; but this is not a security threat because attacks would remain within the same VPN.
Prefix filtering (ASBR-ASBR)? Only IP prefixes are exchanged here, and they define reachability of the PEs. We recommend configuring prefix filters that allow only the required PE IP addresses. (Remember, this is control plane filtering, not data plane.)
ip prefix-list abc deny 0.0.0.0/0 le 32
Route-target filtering (RR-RR)? Each VPN-IPv4 route comes with a set of RTs, defining to which VPN the prefix belongs. Wrongly propagated RTs might lead to intrusions into private VPNs of the other AS.
ip extcommunity-list 101 permit RT:100:3+
No IP routing (RR-RR)? By default, BGP carries the IP address family. This should be switched off.
no bgp default ipv4-unicast
No VPN-IPv4 routing (ASBR-ASBR)? This is not required, and therefore the VPN-IPv4 address family should not be configured.
Measures against prefix flooding (RR-RR and ASBR-ASBR)? To avoid memory exhaustion on an ASBR or RR, a maximum number of prefixes for each BGP peer should be configured.
neighbor 192.0.2.2 maximum-prefix 100 80
Case C does not have VRFs on the ASBR; therefore, the prefix limits per VRF are not applicable here.
Standard router security? Use standard router security features to protect the routers against intrusions and DoS attacks. Receive ACLs (rACLs) or Control Plane Policing (CoPP) are important examples.
Please refer to Chapter 5 for detailed configuration examples.
NOTE
Recommendation
Configure all these security mechanisms as tightly as possible. A hacker only needs one weakness to succeed!
The problem with overlapping RDs also exists in model C. Please refer to the description of model B earlier in this chapter.
NOTE
Recommendation
Uniqueness of RDs across all autonomous systems is essential and must be enforced through operational means. The easiest way is for all service providers involved to follow the conventions to use the AS number as part of the RD. Also, refer to the problem of overlapping RDs in the section "Case B."
In summary, the control plane in model C can be secured well, but this is a complex process, with many important details. However, the main concern at the moment for case C is data plane security.
Also in model C, two types of packets are exchanged between the ASBRs: IP packets and labeled packets. IP packets are used for eBGP between the ASBRs. All other traffic, including the eBGP peering between the RRs is over labeled packets. Therefore, it is recommended, as in case B, to filter all IP traffic with the exception of the required eBGP peering between the ASBRs. This prevents a large number of potential attacks where packets could be sent to routers in the other AS. This secures IP traffic well.
Labeled packets, however, are much harder to control. In case C, there are at least two labels for each packet: the PE label, which defines the LSP to the egress PE, and the VPN label, which defines the VPN on the PE the packet belongs to.
The top label could in theory be checked, but as described previously for model B, this check is currently not implemented for interfaces in the global routing table. This means that a service provider may send packets to any egress PE to which the peering ASBR holds an LSP. It includes at least all PEs which hold Inter-AS VPNs.
In addition, there is a fundamental architectural security issue in model C for which there is currently no solution: because the ASBR routers in this architecture do not hold any VPN information, the VPN label cannot be controlled at the ASBR. This means that it is possible for another service provider to insert packets with anything below the top PE label. Therefore, the security risk of this architectural model is very high: not only can a service provider send packets into any VPN for which it holds a valid PE label, it could also fake a packet such that there is an IP packet directly underneath the top label. This packet would be forwarded to the egress PE with the top label, and before reaching that PE, penultimate hop-popping would remove the top label. If there is an IP packet underneath, this IP packet would be routed in the other service provider's IGP and could thus reach any router in its network.
How easy are attacks of this form?
All of those attacks allow only unidirectional traffic: the return path to the attacker would never be open. However, there have been a number of security issues in the past based on single packets, such as the "ping of death" in 1990 and the "code red" worm in 2001. Thus this issue has to be considered.
Attacks where labeled packets are crafted can only be carried out from a point in the network where labels are accepted: this is normally only the service provider core because all customer connections accept only IP packets. The following section describes the Carrier's Carrier (CsC) case, which allows also labeled packets to enter from a customer, but the CsC architecture does not allow label spoofing: in CsC, the top labels on the data plane are checked. Therefore, the threat of label spoofing can only originate within a service provider network.
Spoofing of the top label can in principle only occur on the ASBR itself because labels have only a local significance to a single router and are swapped at every hop. Every router has a label binding for an incoming label to an outgoing label. To create a binding for a fake outgoing label, one would therefore have to statically configure a binding on the ASBR. This way, an LSP could be constructed hop by hop through the MPLS core. This LSP could be terminated on a packet generator as a source, in which case any type of packet could be generated. There are programs for PCs to generate random packets?also labeled ones.
Even though it is possible to spoof the top label in models B and C, it is not known which label values would lead to which destination. For a targeted attack, an attacker would therefore have to have access the routers in the other AS to check which labels to spoof.
A blind attack is easier, where the entire label space is tried out. A label has 20 bits, which yields one million distinct values. Therefore, by sending one million packets it is possible to test all values. With a minimum packet size of 60 bytes, this means that 60 MB need to be sent. Over a 10 Mbps connection, this would take less than a minute. Over a 128 kbps, it would take a bit more than an hour. Note that label allocation is not random, so in practice the attack would be easier than this. So if only a single packet is needed for an attack, it is quite feasible simply to try all possible label values.
To summarize the various Inter-AS options:
Model A is as secure as standard (single-AS) MPLS as defined in RFC 2547. There are only IP interfaces into the core, providing a strong separation of the outside to the inside of the core. It is suitable for multiprovider MPLS networks, even if the providers do not trust each other.
Model B can be secured well on the control plane, but on the data plane there is a problem currently in that the top label is not checked. This weakness could be exploited to send crafted packets from inside one MPLS core into the other one. It is not possible to establish bidirectional communications, but it is possible to send faked packets into the core of the other provider or VPNs connected there. There is a workaround: a separate ASBR can be provisioned that blocks faked packets. With this workaround, model B can be deployed today in Inter-AS scenarios.
Model C can also be secured well on the control plane, but the data plane has architectural weaknesses: because the ASBRs do not have any VPN information, they cannot check the second label. This enables unidirectional packets to be sent into private VPNs of the other AS, and into the other core directly. This architectural weakness cannot be solved by configuration. As in case B, the top label cannot be checked, and the consequences are the same as in case B as well. Model C can therefore only be deployed where service providers trust each other fully.
NOTE
Recommendation
To implement interprovider VPNs, the most secure option today is model A. We recommend using this model as long as the scalability requirements permit. Model B can be used in an interprovider setup only with some specific security measures put in place. Model C is not suitable for general interprovider VPNs.
As previously mentioned, if several autonomous systems are under one administrative control, all models?even model C?can be deployed today.
Another option for providing a multiprovider scenario is the so-called "hierarchical MPLS," where a carrier buys an MPLS VPN from another carrier and, in turn, offers VPNs to its customers: this is the Carriers' Carrier architecture.
