A virtual private network (VPN) is an extension of an enterprise's private intranet across a public network such as the Internet, creating a secure private connection, essentially through a private tunnel.

Traditional corporate networks were largely self-contained. They were operated by the corporation; they used private facilities (e.g., leased lines or frame relay circuits); they carried only internal traffic; and they were generally inaccessible by outside users. In this closed, tightly controlled environment, the traditional corporate network was usually considered to be secure.

But today's corporate networks are evolving toward a new business model: the extended corporate network is now a collection of physically separated intranets interconnected over the public Internet, and inter-company communication is now a business necessity. This new business model presents security exposures that were not present the traditional corporate network model. Designers and operators of Virtual Private Networks must understand these exposures and protect against them. A well thought out company-wide VPN Security Policy will be critical to assure that your VPN will interconnect a set of intranets--both within a single company and between companies--with security as good as, or better than, that of traditional self-contained, self-administered corporate networks

A 1997 VPN Research Report, by Infonetics Research, Inc., estimates savings from 20% to 47% of wide area network (WAN) costs by replacing leased lines to remote sites with VPNs. And, for remote access VPNs, savings can be 60% to 80% of corporate remote access dial-up costs. Additionally, Internet access is available worldwide where other connectivity alternatives may not be available.

Vendors' VPN offerings can be categorized in a number of ways. In our opinion the most important differentiator is the protocol layer on which the VPN is realized. In this context, there are the following different approaches to VPN implementation:

• Network layer-based (IPSec-based)
• Data link layer-based (layer 2-based)

There are other methods that operate on upper layers and complement a VPN solution, such as SOCKS, Secure Sockets Layer (SSL), or Secure Multipurpose Internet Mail Extension (S-MIME). Some vendors' solutions use only the upper layer protocols to construct a VPN, usually a combination of SOCKS V5 and SSL.

Within the layered communications protocol stack model, the network layer (IP in the case of the TCP/IP stack) is the lowest layer that can provide end-to-end security. Network-layer security protocols provide blanket protection for all upper-layer application data carried in the payload of an IP datagram, without requiring a user to modify the applications.

The solutions are based on the IP Security Architecture (IPSec) open framework, defined by the IPSec Working Group of the IETF. IPSec is called a framework because it provides a stable, long lasting base for providing network layer security. It can accommodate today's cryptographic algorithms, and can also accommodate newer, more powerful algorithms as they become available. IPv6 implementations are required to support IPSec, and IPv4 implementations are strongly recommended to do so. In addition to providing the base security functions for the Internet, IPSec furnishes flexible building blocks from which robust, secure virtual private networks can be constructed.

The IPSec Working Group has concentrated on defining protocols to address several major areas:

• Data origin authentication verifies that each datagram was originated by the claimed sender.
• Data integrity verifies that the contents of the datagram were not changed in transit, either deliberately or due to random errors.
• Data confidentiality conceals the cleartext of a message, typically by using encryption.
• Replay protection assures that an attacker can not intercept a datagram and play it back at some later time without being detected.
• Automated management of cryptographic keys and security associations assures that a company's VPN policy can be conveniently and accurately implemented throughout the extended network with little or no manual configuration. These functions make it possible for a VPN's size to be scaled to whatever size a business requires.

The principal IPSec protocols are:

• IP Authentication Header (AH) provides data origin authentication, data integrity, and replay protection.
• IP Encapsulating Security Payload (ESP) provides data confidentiality, data origin authentication, data integrity, and replay protection.
• Internet Security Association and Key Management Protocol (ISAKMP) provides a method for automatically setting up security associations and managing their cryptographic keys.

The concept of a Security Association (SA) is fundamental to IPSec. An SA is a unidirectional (simplex) logical connection between two IPSec systems, uniquely identified by the following triple:

Security Parameter Index
IP Destination Address
Security Protocol

The definition of the members is as follows:

Security Parameter Index (SPI)

This is a 32-bit value used to identify different SAs with the same destination address and security protocol. The SPI is carried in the header of the security protocol (AH or ESP). The SPI has only local significance, as defined by the creator of the SA. The SPI values in the range 1 to 255 are reserved by the Internet Assigned Numbers Authority (IANA). The SPI value of 0 must be used for local implementation-specific purposes only. Generally the SPI is selected by the destination system during the SA establishment.

IP Destination Address

This address may be a unicast, broadcast or multicast address. However, currently SA management mechanisms are defined only for unicast addresses.

Security Protocol

This can be either AH or ESP.

An SA can be in either of two modes: transport or tunnel, depending on the mode of the protocol in that SA. You can find the explanation of these protocol modes later in this chapter.

Because SAs are simplex, for bidirectional communication between two IPSec systems, there must be two SAs defined, one in each direction.

An SA gives security services to the traffic carried by it either by using AH or ESP, but not both. In other words, for a connection that should be protected by both AH and ESP, two SAs must be defined for each direction. In this case, the set of SAs that define the connection is referred to as an SA bundle. The SAs in the bundle do not have to terminate at the same endpoint. For example, a mobile host could use an AH SA between itself and a firewall and a nested ESP SA that extends to a host behind the firewall.

An IPSec implementation maintains two databases related to SAs:

Security Policy Database (SPD)

The Security Policy Database specifies what security services are to be offered to the IP traffic, depending on factors such as source, destination, whether it is inbound, outbound, etc. It contains an ordered list of policy entries, separate for inbound and or outbound traffic. These entries might specify that some traffic must not go through IPSec processing, some must be discarded and the rest must be processed by the IPSec module. Entries in this database are similar to the firewall rules or packet filters.

Security Association Database (SAD)

The Security Association Database contains parameter information about each SA, such as AH or ESP algorithms and keys, sequence numbers, protocol mode and SA lifetime. For outbound processing, an SPD entry points to an entry in the SAD. That is, the SPD determines which SA is to be used for a given packet. For inbound processing, the SAD is consulted to determine how the packet must be processed.

Tunneling

In general tunneling is used to carry traffic of one protocol over a network that does not support that protocol directly. For example, NetBIOS or IPX can be encapsulated in IP to carry it over a TCP/IP WAN link. In the case of IPSec, IP is tunneled through IP for a slightly different purpose: to provide total protection, including the header of the encapsulated packet. If the encapsulated packet is encrypted, an intruder cannot figure out for example the destination address of that packet. (Without tunneling he or she could.) The internal structure of a private network can be concealed in this way.

AH is used to provide integrity and authentication to IP datagrams. Optional replay protection is also possible. Although its usage is optional, the replay protection service must be implemented by any IPSec-compliant system. The mentioned services are connectionless, that is they work on a per-packet basis.

AH authenticates as much of the IP datagram as possible. Some fields in the IP header change en-route and their value cannot be predicted by the receiver. These fields are called mutable and are not protected:

When protection of these fields is required, tunneling should be used. The payload of the IP packet is considered immutable and is always protected by AH.

AH is identified by protocol number 51, assigned by the IANA. The protocol header (IPv4, IPv6, or Extension) immediately preceding the AH header contains this value in its Protocol (IPv4) or Next Header (IPv6, Extension) field.

AH processing is applied only to non-fragmented IP packets. However an IP packet with AH applied can be fragmented by intermediate routers. In this case the destination first reassembles the packet and then applies AH processing to it. If an IP packet that appears to be a fragment (offset field is non-zero, or the More Fragments bit is set) is input to AH processing, it is discarded.

AH Format

Next Header

The Next Header is an 8-bit field that identifies the type of the next payload after the Authentication Header. The value of this field is chosen from the set of IP protocol numbers defined in the most recent "Assigned Numbers" RFC from the Internet Assigned Numbers Authority (IANA).

Payload Length

This field is 8 bits long and contains the length of the AH header expressed in 32-bit words, minus 2. It does not relate to the actual payload length of the IP packet as a whole. If default options are used, the value is 4. (Three 32-bit fixed words plus three 32-bit words of authentication data minus two.)

Reserved

This field is reserved for future use. Its length is 16 bits and it is set to zero.

Security Parameter Index (SPI)

This field is 32 bits in length.

Sequence Number

This 32-bit field is a monotonically increasing counter which is used for replay protection. Replay protection is optional; however, this field is mandatory. The sender always includes this field and it is at the discretion of the receiver to process it or not. At the establishment of an SA the sequence number is initialized to zero. The first packet transmitted using the SA has a sequence number of 1. Sequence numbers are not allowed to repeat. Thus the maximum number of IP packets that can be transmitted on any given SA is 2³²-1. After the highest sequence number is used, a new SA and consequently a new key is established. Anti-replay is enabled at the sender by default. If upon SA establishment the receiver chooses not to use it, the sender does not concern with the value in this field anymore.

Authentication Data

This is a variable-length field, also called Integrity Check Value (ICV). The ICV for the packet is calculated with the algorithm selected at the SA initialization. The authentication data length is an integral multiple of 32 bits. As its name tells, it is used by the receiver to verify the integrity of the incoming packet.

When doing the ICV calculation, the mutable fields are considered to be filled with zero.

AH can be used in two ways: transport mode and tunnel mode

 

 

In transport mode the original IP datagram is taken and the AH header is inserted right after the IP header. If the datagram already has IPSec header(s), then the AH header is inserted before any of those.

The transport mode is used by hosts, not by gateways. Gateways are not even required to support transport mode.

The advantage of the transport mode is less processing overhead. The disadvantage is that the mutable fields are not authenticated.

With tunnel mode the tunneling concept is applied a new IP datagram is constructed and the original IP datagram is made the payload of it. Then AH in transport mode is applied to the resulting datagram.

The tunnel mode is used whenever either end of a security association is a gateway. Thus, between two firewalls the tunnel mode is always used.

Although gateways are supposed to support tunnel mode only, often they can also work in transport mode. This mode is allowed when the gateway acts as a host, that is in cases when traffic is destined to itself.

In tunnel mode the outer headers' IP addresses does not need to be the same as the inner headers' addresses. For example two security gateways may operate an AH tunnel which is used to authenticate all traffic between the networks they connect together. This is a very typical mode of operation. Hosts are not required to support tunnel mode, but often they do.

The advantages of the tunnel mode are total protection of the encapsulated IP datagram and the possibility of using private addresses. However, there is an extra processing overhead associated with this mode.

AH is an integral part of IPv6. In an IPv6 environment, AH is considered an end-to-end payload and it appears after hop-by-hop, routing, and fragmentation extension headers. The destination options extension header(s) could appear either before or after the AH header.

ESP is used to provide integrity check, authentication and encryption to IP datagrams. Optional replay protection is also possible. These services are connectionless, they operate on a per-packet basis. The set of desired services are selectable upon SA establishment. However, some restrictions apply:

• Integrity check and authentication go together
• Replay protection is selectable only with integrity check and authentication
• Replay protection can be selected only by the receiver
• Encryptyon

Encryption is selectable independent of the other services. It is highly recommended that if encryption is enabled, then integrity check and authentication be turned on. If only encryption is used, intruders could forge packets in order to mount cryptanalytic attacks. This is infeasible when integrity check and authentication are in place.

Although both authentication (with integrity check) and encryption are optional, at least one of them is always selected. Otherwise it really does not make sense to use ESP at all.

ESP is identified by protocol number 50, assigned by the IANA. The protocol header (IPv4, IPv6, or Extension) immediately preceding the AH header will contain this value in its Protocol (IPv4) or Next Header (IPv6, Extension) field.

ESP processing is applied only to non-fragmented IP packets. However an IP packet with ESP applied can be fragmented by intermediate routers. In this case the destination first reassembles the packet and then applies ESP processing to it

If both encryption and authentication with integrity check are selected, then the receiver first authenticates the packet and only if this step was successful proceeds with decryption. This mode of operation saves computing resources and reduces the vulnerability to denial of service attacks.

ESP format

Security Parameter Index (SPI)

This field is 32 bits in length.

Sequence Number

This 32-bit field is a monotonically increasing counter. Same as in AH.

Payload Data

The Payload Data field is mandatory. It consists of a variable number of bytes of data described by the Next Header field. This field is encrypted with the cryptographic algorithm selected during SA establishment.

Padding

Most encryption algorithms require that the input data must be an integral number of blocks. Also, the resulting ciphertext (including the Padding, Pad Length and Next Header fields) must terminate on a 4-byte boundary, so that Next Header field is right aligned. That's why this variable length field is included. It can be used to hide the length of the original messages too. However, this could adversely impact the effective bandwidth. Padding is an optional field.

Pad Length

This 8-bit field contains the number of the preceding padding bytes. It is always present, and the value of 0 indicates no padding.

Next Header

The Next Header is an 8-bit mandatory field that shows the data type carried in the payload, for example an upper-level protocol identifier such as TCP. The values are chosen from the set of IP Protocol Numbers defined by the IANA.

Authentication Data

This field is variable in length and contains the ICV calculated for the ESP packet from the SPI to the Next Header field inclusive. The Authentication Data field is optional. It is included only when integrity check and authentication have been selected at SA initialization time.

The ESP specifications require two authentication algorithms to be supported: HMAC with MD5 and HMAC with SHA-1. Often the simpler keyed versions are also supported by the IPSec implementations.

 

Like AH, ESP can be used in two ways: transport mode and tunnel mode.

In transport mode the original IP datagram is taken and the ESP header is inserted right after the IP header. If the datagram already has IPSec header(s), then the ESP header is inserted before any of those. The ESP trailer and the optional authentication data are appended to the payload.

ESP in transport mode provides neither authentication nor encryption for the IP header. This is a disadvantage, since false packets might be delivered for ESP processing. The advantage of transport mode the lower processing overhead.

As in the case of AH, ESP in transport mode is used by hosts, not gateways. Gateways are not even required to support transport mode.

Tunnel mode applies the tunneling principle. A new IP packet is constructed with a new IP header and then ESP in transport mode is applied. Since the original datagram becomes the payload data for the new ESP packet, its protection is total if both encryption and authentication are selected. However, the new IP header is still not protected.

The tunnel mode is used whenever either end of a security association is a gateway. Thus, between two firewalls the tunnel mode is always used.

Although gateways are supposed to support tunnel mode only, often they can also work in transport mode. This mode is allowed when the gateway acts as a host, that is in cases when traffic is destined to itself.

In tunnel mode the outer headers' IP addresses does not need to be the same as the inner headers' addresses. For example two security gateways may operate an ESP tunnel which is used to secure all traffic between the networks they connect together. Hosts are not required to support tunnel mode, but often they do.

The advantages of the tunnel mode are total protection of the encapsulated IP datagram and the possibility of using private addresses. However, there is an extra processing overhead associated with this mode.

Like AH, ESP is an integral part of IPv6. In an IPv6 environment, ESP is considered an end-to-end payload and it appears after hop-by-hop, routing, and fragmentation extension headers. The destination options extension header(s) could appear either before or after the AH header.

Sequence of Actions (outbound):

1. SA lookup
2. Packet encryption
3. Seq. No generation
4. ICV calculation
5. Fragmentation

Sequence of Actions (inbound):

1. Reassembly
2. SA lookup
3. Seq. No verification
4. ICV calculation and verification
5. Packet decryption

Knowing about the security services of ESP, one might ask if there is really a requirement for AH. Why does ESP authentication not cover the IP header as well? There is no official answer to these questions, but here are some points that justify the existence of two different IPSec authentication protocols:

• ESP requires strong cryptographic algorithms to be implemented, whether it will actually be used or not. Strong cryptography is an over-hyped and sensitive topic in some countries, with restrictive regulations in place. It might be troublesome to deploy ESP-based solutions in such areas. However, authentication is not regulated and AH can be used freely around the world.
• Often only authentication is needed. While ESP could have been specified to cover the IP header as well, AH is more performant compared to ESP with authentication only, because of the simpler format and lower processing overhead. It makes sense to use AH in these cases.
• Having two different protocols means finer-grade control over an IPSec network and more flexible security options. By nesting AH and ESP for example, one can implement IPSec tunnels that combine the strengths of both protocols.

The AH and ESP protocols can be applied alone or in combination. Given the two modes of each protocol, there is quite a number of possible combinations. To make things even worse, the AH and ESP SAs do not need to have identical endpoints, so the picture becomes rather complicated. Luckily, out of the many possibilities only a few make sense in real-world scenarios.

There are two approaches for an SA bundle creation:

• Transport adjacency: Both security protocols are applied in transport mode to the same IP datagram. This method is practical for only one level of combination.
• Iterated (nested) tunneling: The security protocols are applied in tunnel mode in sequence. After each application a new IP datagram is created and the next protocol is applied to it. This method has no limit in the nesting levels. However, more than three levels are inpractical.

These approaches can be combined, for example an IP packet with transport adjacency IPSec headers can be sent through nested tunnels.

The practical principle of the combined usage is that upon the receipt of a packet with both protocol headers, the IPSec processing sequence should be authentication followed by decryption. It is a common sense decision not to bother with the decryption of packets of uncertain origin.

Following the above principle, the sender first applies ESP and then AH to the outbound traffic. In fact this sequence is an explicit requirement for transport mode IPSec processing. When using both ESP and AH, a new question arises: should ESP authentication be turned on? AH authenticates the packet anyway. The answer is simple. Turning ESP authentication on makes sense only when the ESP SA extends beyond the AH SA, as in the case of the supplier scenario. In this case, not only does it make sense to use ESP authentication, but it is highly recommended to do so, to avoid spoofing attacks in the intranet.

As far as the modes are concerned, the usual way is that transport mode is used between the endpoints of a connection and tunnel mode is used between two machines when at least one of them is a gateway.