How to Securely Connect On-Prem Infrastructure to Azure and AWS

Connecting an on-premises network to Azure or AWS is relatively straightforward.

Making that connection secure, predictable and supportable is where the real work starts.

A site-to-site VPN can have traffic flowing in an afternoon. ExpressRoute and AWS Direct Connect can provide much more predictable connectivity for production workloads. But none of those technologies solve the underlying design problems for you.

You still need to get the fundamentals right:

• IP addressing
• routing
• BGP
• MTU
• firewall policy
• network segmentation
• redundancy
• monitoring
• cost

Get one of those wrong and the result can be a connection that appears to work but fails as soon as you move a real application across it.

This guide looks at the practical side of connecting on-premises infrastructure to Microsoft Azure and Amazon Web Services (AWS), with particular attention to the problems that tend to appear after the tunnel or circuit is already operational.

Start with the network design, not the VPN

Before choosing a connectivity method, document the networks on both sides.

For example:

On-premises
10.10.0.0/16
10.20.0.0/16
10.30.0.0/16

Azure
10.100.0.0/16

AWS
10.200.0.0/16

That looks simple.

Now imagine the Azure team creates:

10.10.0.0/16

because it was available in their address plan.

You now have overlapping address space.

A packet destined for 10.10.20.50 could legitimately belong to the on-premises network or the Azure VNet. Routing cannot magically resolve that ambiguity.

Overlapping address space causes real problems

This is one of the first things to check before building a hybrid connection.

Document:

• every on-premises subnet
• every Azure VNet and subnet
• every AWS VPC and subnet
• VPN tunnel networks
• management networks
• transit networks
• existing routes
• future address allocations

Do this before ordering circuits or configuring BGP.

If you are already dealing with overlapping networks, options such as NAT or application-level proxying may help in specific scenarios, but they add complexity. Renumbering is painful, but sometimes it is the cleaner long-term solution.

The worst time to discover overlapping address space is after an application migration has already started.

The three main connectivity options

There are three common ways to connect on-premises infrastructure to Azure or AWS.

A production environment can also use more than one.

For example:

                        On-premises
                             |
        +--------------------+--------------------+
        |                    |                    |
   ExpressRoute         VPN tunnels         Direct Connect
   (primary)             (backup)            (primary)
        |                    |                    |
        v                    v                    v
    Azure VNet      Azure VNet / AWS VPC      AWS VPC

The VPN connections can provide a backup path while ExpressRoute or Direct Connect carries normal production traffic.

The important point is that redundancy needs to be designed into the routing, not simply provided by having two physical connections.

Site-to-site VPN

A site-to-site VPN creates an encrypted IPsec connection between an on-premises firewall or router and a cloud VPN gateway.

This is normally the easiest way to establish initial connectivity.

It is useful for:

• development and testing
• smaller environments
• initial cloud migrations
• branch offices
• temporary migration networks
• backup connectivity
• environments where dedicated private connectivity isn’t justified

AWS Site-to-Site VPN provides two tunnels for a connection, allowing traffic to fail over between tunnel endpoints. Standard AWS VPN tunnels support up to 1.25 Gbps per tunnel. AWS also supports larger tunnel bandwidth options when attached to a Transit Gateway or Cloud WAN. The current figures are in the AWS Site-to-Site VPN quotas.

Azure VPN Gateway performance depends on the gateway SKU. Current SKUs range from hundreds of Mbps to multi-gigabit aggregate throughput, with the actual performance depending on SKU, generation, algorithms and traffic characteristics. Microsoft publishes the per-SKU figures in About Azure VPN Gateway.

That distinction matters.

If you have a 2 Gbps internet connection, it does not mean your Azure VPN gateway or AWS VPN tunnel can automatically push 2 Gbps of application traffic.

Build two tunnels

For production connectivity, don’t design around a single IPsec tunnel if the cloud platform provides redundant tunnel endpoints.

At a minimum, document:

On-premises firewall
       |
       +------ Tunnel 1 ------ Cloud gateway
       |
       +------ Tunnel 2 ------ Cloud gateway

Then decide which path should normally be preferred.

With BGP, this can be controlled using route attributes such as local preference and AS path manipulation. With static routing, failover requires a different mechanism.

The important thing is to test the failure.

Don’t simply confirm that both tunnels show UP.

Disconnect one and verify that application traffic actually follows the surviving path.

Azure VPN Gateway gotchas

Azure requires a subnet named:

GatewaySubnet

for a virtual network gateway.

Microsoft currently recommends /27 or larger for the gateway subnet. A /29 is possible in limited configurations, but using /27 provides considerably more room for future gateway requirements. You also should not place workloads in the gateway subnet.

A typical design might therefore look like:

VNet: 10.100.0.0/16

GatewaySubnet: 10.100.255.0/27
Servers:       10.100.10.0/24
Applications:  10.100.20.0/24
Management:    10.100.30.0/24

The subnet is not named AzureGatewaySubnet.

It must be:

GatewaySubnet

That sounds trivial, but it is exactly the sort of detail that can stop a deployment.

You can create the subnet with Azure CLI:

az network vnet subnet create \
  --resource-group rg-network \
  --vnet-name vnet-production \
  --name GatewaySubnet \
  --address-prefixes 10.100.255.0/27

Microsoft also warns against putting an NSG on the gateway subnet.

Azure BGP prefix limits

Azure VPN Gateway supports a maximum number of prefixes per BGP peer. If the on-premises device advertises more than the supported number, Azure drops the BGP session.

The current limit is documented in About BGP with Azure VPN Gateway, and it is worth checking against your own environment rather than assuming a figure from an article.

Either way, the practical advice is the same: don’t advertise your entire corporate routing table into Azure just because BGP makes it easy.

Advertise what Azure actually needs.

For example:

10.10.0.0/16
10.20.0.0/16
10.30.0.0/16

is much easier to understand and troubleshoot than hundreds of individual branch prefixes.

There is also a related restriction worth knowing: Microsoft documents that routes advertised from on-premises which are identical to a VNet prefix are rejected. If you are advertising a large aggregate such as 10.0.0.0/8 from on-premises and your VNet sits inside that range, check how the overlap will be handled before building the peering.

AWS Site-to-Site VPN gotchas

AWS also uses BGP or static routing depending on the design.

For a production environment, BGP is generally more useful because it allows routes to change dynamically when a tunnel or path fails.

AWS provides configuration information for each tunnel, including the parameters required by your customer gateway device.

The AWS CLI can be used to inspect the connection:

aws ec2 describe-vpn-connections \
  --vpn-connection-ids vpn-0123456789abcdef0

This is useful when troubleshooting whether the problem is:

• tunnel establishment
• BGP
• routing
• encryption parameters
• the customer gateway configuration

AWS also has a useful distinction between the VPN tunnel bandwidth and the bandwidth of the underlying internet connection. A standard tunnel is limited to up to 1.25 Gbps, regardless of how much bandwidth exists on the physical internet circuit.

ExpressRoute

Azure ExpressRoute provides private connectivity between your network and Microsoft.

The important point is that private does not mean automatically encrypted.

Microsoft provides MACsec encryption for ExpressRoute Direct, which encrypts traffic at Layer 2 between the customer’s network equipment and Microsoft’s edge routers. MACsec is available specifically for ExpressRoute Direct rather than provider-provisioned ExpressRoute circuits.

So instead of saying:

ExpressRoute is private, therefore our traffic doesn’t need encryption.

the design question should be:

What protection does this traffic require, and where is that protection provided?

That is a much better security discussion.

ExpressRoute MTU

ExpressRoute uses a maximum IP MTU of 1500 bytes. Microsoft documents this in the ExpressRoute routing requirements.

That becomes important when an on-premises network normally uses jumbo frames.

For example, you might have:

Server
  |
  | MTU 9000
  |
Switch
  |
Router
  |
ExpressRoute
  |
Azure

The fact that the local network supports jumbo frames doesn’t mean the ExpressRoute path does.

If an application starts behaving strangely after migration, MTU should be on the troubleshooting list.

AWS Direct Connect

AWS Direct Connect provides a private connection between your network and an AWS Direct Connect location.

There are two important connection types to distinguish.

Dedicated Direct Connect

Dedicated connections are ordered directly from AWS and are available at fixed port speeds, currently 1 Gbps, 10 Gbps, 100 Gbps and 400 Gbps.

Hosted Direct Connect

Hosted connections are supplied through an AWS Direct Connect Partner and are available at a much wider range of speeds, from 50 Mbps up to 25 Gbps depending on the partner and location.

You will often see Direct Connect described as ranging from “50 Mbps to 100 Gbps”. That figure mixes the two connection types together and isn’t a range you can actually order from a single product.

When comparing bandwidth options, establish which connection type you are actually buying. The current speeds are listed in the AWS Direct Connect documentation.

Direct Connect isn’t automatically encrypted

Like ExpressRoute, Direct Connect provides private connectivity but that does not automatically mean application traffic is encrypted end to end.

If the security requirement calls for encryption, you can use encryption over the private connection, including appropriate VPN designs.

The security model should be based on the sensitivity of the traffic, not simply on whether the packets traverse the public internet.

Direct Connect BGP prefix limits

This is one of the Direct Connect limits that can catch out a large environment.

AWS applies an inbound prefix limit to private virtual interfaces, with a documented default allocation and the ability to request a larger allocation under supported configurations. If the number of advertised prefixes exceeds the allocation, the BGP session goes down.

The current values are in the AWS Direct Connect quotas.

The practical lesson is the same:

Don’t advertise everything just because you can.

Aggregate routes where possible and explicitly control what is advertised towards AWS.

For example:

Instead of:

10.10.1.0/24
10.10.2.0/24
10.10.3.0/24
10.10.4.0/24
...

Prefer an aggregate where routing allows:

10.10.0.0/16

That also makes the resulting routing table much easier to understand.

Direct Connect MTU

AWS Direct Connect private virtual interfaces support an MTU of either 1500 or 9001. Transit virtual interfaces support 1500 or 8500.

Jumbo frames need to be supported throughout the relevant path, and enabling them can affect the underlying connection.

This is very different from ExpressRoute, where the MTU is 1500.

Don’t assume that because your AWS EC2 instances support jumbo frames, the entire hybrid path does.

BGP is where the design becomes interesting

BGP is useful because it lets the network react to changes instead of relying entirely on static routes.

But BGP introduces another set of failure modes.

At minimum, document:

On-prem ASN
Cloud ASN
BGP peer IP
Advertised prefixes
Received prefixes
Local preference
AS path
MED
Route filtering
Failover preference

For example:

                 Primary
On-prem -------- ExpressRoute -------- Azure
   |
   | Backup
   +---------- VPN ------------------- Azure

You might want Azure routes learned over ExpressRoute to be preferred over the same routes learned through the VPN.

That preference needs to be explicit.

Otherwise you can end up with asymmetric routing:

Request:
On-prem -> ExpressRoute -> Azure

Response:
Azure -> VPN -> On-prem

Some applications tolerate this.

Some firewalls absolutely do not. A stateful firewall that sees the return traffic without having seen the request will usually drop it, and the resulting symptom looks like intermittent application failure rather than a routing problem.

ASN conflicts

Don’t randomly select an ASN because the field is asking for one.

Azure reserves several ASNs for its own use, including private ASNs in the 65515 to 65520 range. The Azure VPN Gateway documentation also lists other reserved public and IANA ranges.

ExpressRoute uses AS 12076 on the Microsoft side for Azure private and Microsoft peering.

The practical approach is to choose your enterprise ASN deliberately and document it before building the BGP sessions.

For example:

Organisation ASN: 65010

Azure:
Microsoft ASN: 12076
On-prem peer ASN: 65010

AWS:
AWS-side ASN: documented by the selected gateway/VIF
On-prem peer ASN: 65010

The exact AWS ASN depends on the AWS connectivity design, so use the generated AWS configuration rather than assuming the same value applies to every service.

MTU is one of the first things to test

MTU problems are particularly nasty because basic connectivity can still work.

You might be able to:

ping 10.100.10.10

while an application fails when it sends larger packets.

On Windows, test with:

ping -f -l 1472 10.100.10.10

The 1472 payload plus the 8-byte ICMP header and 20-byte IPv4 header equals a 1500-byte packet.

If that fails, reduce the payload:

ping -f -l 1400 10.100.10.10

On Linux:

ping -M do -s 1472 10.100.10.10

Then work down until you identify the largest packet that successfully crosses the path without fragmentation.

For VPN connections, remember that IPsec adds overhead. The usable MTU can therefore be lower than 1500, and TCP MSS adjustment may be required depending on the VPN implementation.

This is one reason an application can work perfectly on the LAN but behave badly across a hybrid connection.

Don’t forget firewall policy

A hybrid connection is not a reason to allow everything between the networks.

Avoid rules such as:

On-premises -> Azure
Source: Any
Destination: Any
Service: Any
Action: Allow

Instead, define the actual communication requirements.

For example:

On-prem application subnet
        |
        | TCP 443
        v
Azure application subnet

Azure backup subnet
        |
        | TCP 443 / required backup ports
        v
On-prem backup infrastructure

The routing layer decides where traffic can go.

The firewall decides what traffic is allowed to go there.

Those are different controls.

Network segmentation matters more after the connection works

A common mistake is to treat the hybrid connection as one large extension of the LAN.

Instead, think about trust zones:

                 On-premises
                     |
                 Firewall
                     |
              Hybrid transit
                /          \
               /            \
          Azure              AWS
          /   \              /   \
     Web VNet App VNet    Web VPC App VPC

A compromised cloud workload should not automatically have a route to every server on-premises.

Likewise, an infected workstation should not automatically be able to reach every workload in Azure and AWS.

Use:

• separate subnets
• route filtering
• firewalls
• security groups
• NSGs
• network ACLs where appropriate
• private endpoints
• application-level controls

The hybrid connection should provide the paths that are required, not become a giant trusted pipe.

Monitoring the connection

A green VPN tunnel doesn’t mean the application path is healthy.

Monitor at several layers.

VPN

Check:

• tunnel state
• IKE status
• IPsec security associations
• packet counters
• rekey events
• tunnel flaps

BGP

Check:

• session state
• prefixes received
• prefixes advertised
• route changes
• session resets

On Cisco IOS:

show ip bgp summary
show ip route
show ip bgp

On a FortiGate:

get router info bgp summary
get router info routing-table all

Azure

Azure VPN diagnostics can help identify BGP disconnects and route problems. Microsoft documents checking BGP prefix counts and gateway route propagation when troubleshooting VPN Gateway.

Azure CLI can be used to inspect a connection:

az network vpn-connection show \
  --resource-group rg-network \
  --name connection-onpremises

AWS

AWS CLI:

aws ec2 describe-vpn-connections \
  --vpn-connection-ids vpn-0123456789abcdef0

For Direct Connect:

aws directconnect describe-virtual-interfaces

and:

aws directconnect describe-direct-connect-gateway-attachments

These commands won’t replace packet captures or router diagnostics, but they help establish whether the cloud service believes the connection and routing objects are healthy.

Cost is part of the architecture

VPN is not automatically the right answer because it is cheaper.

ExpressRoute and Direct Connect introduce additional costs that can include:

• port or circuit charges
• connectivity-provider charges
• cross-region traffic
• data transfer
• cloud gateway charges
• VPN backup costs
• redundant circuits

The exact price depends on region, provider, capacity and traffic pattern, so the connectivity decision should be based on the actual traffic requirements rather than a generic “Direct Connect is expensive” assumption.

For many organisations, the interesting calculation is not:

How much does the circuit cost?

It is:

What does an hour of application downtime, degraded performance or failed migration cost?

That can change the architecture decision considerably.

A practical production design

For a production environment with meaningful on-premises dependencies, a reasonable starting architecture is:

                        On-premises
                             |
             +---------------+---------------+
             |                               |
      ExpressRoute                    Direct Connect
       (primary)                         (primary)
             |                               |
             |         VPN backup paths      |
             |     +---------------------+   |
             |     |                     |   |
             v     v                     v   v
         Azure VNet                    AWS VPC

The exact topology will depend on whether Azure and AWS need to communicate with each other, whether a central transit architecture is being used, and whether the on-premises network already has redundant WAN connectivity.

The important characteristics are:

• no overlapping address space
• controlled route advertisements
• BGP where dynamic routing is useful
• redundant paths
• explicit primary and backup preferences
• firewall inspection
• network segmentation
• documented MTU
• monitoring
• tested failover

Test the failure, not just the installation

A hybrid connection isn’t finished when:

VPN: UP
BGP: ESTABLISHED

appears on the screen.

Test what happens when:

1. Primary VPN tunnel fails.
2. Secondary VPN tunnel fails.
3. BGP session drops.
4. Primary circuit fails.
5. Firewall fails over.
6. A route disappears.
7. A route is accidentally advertised.
8. MTU is reduced.
9. One cloud subnet becomes unreachable.
10. The on-premises WAN fails.

During testing, measure:

• failover time
• packet loss
• route convergence
• application recovery
• firewall state
• monitoring alerts

A connection that fails over in theory but causes a five-minute application outage may not meet the business requirement.

The hybrid connectivity checklist

Before putting a production hybrid connection into service, check:

• Azure and AWS address spaces do not overlap with on-premises.
• Cloud subnet allocations are documented.
• The Azure GatewaySubnet is correctly sized.
• BGP ASNs are documented and valid.
• Only required prefixes are advertised.
• BGP failover behaviour has been tested.
• MTU has been tested end to end.
• Firewall rules are restricted to required traffic.
• Cloud workloads are segmented.
• VPN tunnels or private circuits have redundancy.
• Monitoring covers tunnels, BGP and traffic.
• Route changes generate alerts.
• Backup connectivity has been tested.
• Cloud egress and connectivity costs are understood.
• The failure procedure is documented.

Build the network around failure

The technology is only one part of hybrid connectivity.

A VPN can establish a secure tunnel in a relatively short time. ExpressRoute and Direct Connect can provide private connectivity for larger production environments. But the connection itself doesn’t solve overlapping networks, route leaks, MTU problems, BGP limits or poorly designed firewall rules.

Those are the problems that determine whether the hybrid network is actually supportable.

Start with the address plan.

Then design the routes.

Then decide how the traffic should be protected and inspected.

Then build redundancy.

Finally, test the failure scenarios before the application depends on the connection.

That’s the difference between having a cloud connection and having a hybrid network you can actually operate.

Daniel Lutton

From my early days on the helpdesk through roles as a service desk manager, systems administrator, and network engineer, I’ve spent more than 25 years in the IT world. As I transition into cyber security, my goal is to make tech a little less confusing by sharing what I’ve learned and helping others wherever I can.