Connect On-Prem Infrastructure to Cloud

Connecting on-prem infrastructure to Azure or AWS is relatively straightforward. Making that connection secure, predictable and supportable is where the real work starts. A site-to-site VPN can have traffic flowing in an afternoon, and ExpressRoute and AWS Direct Connect can provide much more predictable connectivity for production workloads. But none of those technologies solve the underlying design problems for you.

You still need to get the fundamentals right: IP addressing, routing, BGP, MTU, firewall policy, network segmentation, redundancy, monitoring and cost. Get one of those wrong and the result can be a connection that appears to work, then fails as soon as you move a real application across it.

This guide covers the practical side of connecting on-premises infrastructure to Microsoft Azure and Amazon Web Services (AWS), with particular attention to the problems that tend to appear after the tunnel or circuit is already up.

Start with the network design, not the VPN

Before choosing a connectivity method, document the networks on both sides. For example:

text

On-premises
10.10.0.0/16
10.20.0.0/16
10.30.0.0/16

Azure
10.100.0.0/16

AWS
10.200.0.0/16

That looks simple. Now imagine the Azure team creates 10.10.0.0/16 because it was free in their own address plan. You now have overlapping address space. A packet destined for 10.10.20.50 could legitimately belong to the on-premises network or to the Azure VNet, and routing can’t resolve that ambiguity for you.

Overlapping address space causes real problems

This is one of the first things to check before building a hybrid connection. Document every on-premises subnet, every Azure VNet and subnet, every AWS VPC and subnet, the VPN tunnel, management and transit networks, existing routes and future address allocations. Do it before ordering circuits or configuring BGP.

If you’re already dealing with overlapping networks, NAT or application-level proxying may help in specific scenarios, but both add complexity. Renumbering is painful, but sometimes it’s the cleaner long-term answer. The worst time to discover overlapping address space is after an application migration has started.

The three main connectivity options

There are three common ways to connect on-premises infrastructure to Azure or AWS:

OptionTypical useMain advantageMain limitation
Site-to-site VPNSmaller environments, migrations, backup pathsFast and comparatively inexpensiveInternet-dependent
Azure ExpressRouteProduction Azure connectivityPrivate connectivity and predictable routingMore complex and costly
AWS Direct ConnectProduction AWS connectivityPrivate connectivity and predictable routingMore complex and costly

A production environment can use more than one at the same time:

text

                        On-premises
                             |
        +--------------------+--------------------+
        |                    |                    |
   ExpressRoute         VPN tunnels         Direct Connect
   (primary)             (backup)            (primary)
        |                    |                    |
        v                    v                    v
    Azure VNet      Azure VNet / AWS VPC      AWS VPC

Here the VPN connections provide a backup path while ExpressRoute and Direct Connect carry normal production traffic. The important point is that redundancy has to be designed into the routing. Having two physical connections isn’t enough on its own.

Site-to-site VPN

A site-to-site VPN creates an encrypted IPsec connection between an on-premises firewall or router and a cloud VPN gateway. It’s normally the easiest way to establish initial connectivity, and it suits development and testing, smaller environments, initial cloud migrations, branch offices, temporary migration networks, backup connectivity, and anywhere dedicated private connectivity isn’t justified.

AWS Site-to-Site VPN provides two tunnels per connection, so traffic can fail over between tunnel endpoints. Standard tunnels support up to 1.25 Gbps each, and AWS offers larger tunnel bandwidth options when the VPN attaches to a Transit Gateway or Cloud WAN. The current figures are in the AWS Site-to-Site VPN quotas.

Azure VPN Gateway performance depends on the gateway SKU. Current SKUs range from hundreds of Mbps to multi-gigabit aggregate throughput, and actual performance depends on the SKU, generation, algorithms and traffic. Microsoft publishes the per-SKU figures in About Azure VPN Gateway.

That distinction matters. A 2 Gbps internet connection doesn’t mean your Azure VPN gateway or AWS VPN tunnel can push 2 Gbps of application traffic.

Build two tunnels

For production connectivity, don’t design around a single IPsec tunnel when the cloud platform provides redundant tunnel endpoints:

text

On-premises firewall
       |
       +------ Tunnel 1 ------ Cloud gateway
       |
       +------ Tunnel 2 ------ Cloud gateway

Then decide which path should normally be preferred. With BGP, you can control that with route attributes such as local preference and AS path prepending. With static routing, failover needs a different mechanism.

Most importantly, test the failure. Don’t just confirm both tunnels show UP. Disconnect one and verify that application traffic actually follows the surviving path.

Azure VPN Gateway gotchas

Azure requires a subnet named exactly GatewaySubnet for a virtual network gateway. Not AzureGatewaySubnet, just GatewaySubnet. That sounds trivial, but it’s exactly the sort of detail that stops a deployment.

Microsoft currently recommends a /27 or larger for the gateway subnet. A /29 is possible in limited configurations, but a /27 leaves considerably more room for future gateway requirements. Don’t place workloads in the gateway subnet, and Microsoft also warns against putting an NSG on it. A typical design looks like this:

text

VNet: 10.100.0.0/16

GatewaySubnet: 10.100.255.0/27
Servers:       10.100.10.0/24
Applications:  10.100.20.0/24
Management:    10.100.30.0/24

You can create the subnet with the Azure CLI:

bash

az network vnet subnet create \
  --resource-group rg-network \
  --vnet-name vnet-production \
  --name GatewaySubnet \
  --address-prefixes 10.100.255.0/27

Azure BGP prefix rules

Azure VPN Gateway supports a maximum number of prefixes per BGP peer, and if the on-premises device advertises more than that, Azure drops the BGP session. Check the current limit against your own environment rather than assuming a figure from an article. Either way, the practical advice is the same: don’t advertise your entire corporate routing table into Azure just because BGP makes it easy. Advertise what Azure actually needs. A short list like this is much easier to understand and troubleshoot than hundreds of individual branch prefixes:

text

10.10.0.0/16
10.20.0.0/16
10.30.0.0/16

There’s a related restriction worth knowing. According to Microsoft’s BGP overview for Azure VPN Gateway, you can advertise large aggregates that include your VNet ranges, such as 10.0.0.0/8, but any route identical to a VNet prefix is rejected. Check how that will interact with your aggregates before building the peering.

AWS Site-to-Site VPN gotchas

AWS also supports either BGP or static routing. For production, BGP is generally more useful, because routes can change dynamically when a tunnel or path fails. AWS provides configuration details for each tunnel, including the parameters your customer gateway device needs, and you can inspect the connection with the CLI:

bash

aws ec2 describe-vpn-connections \
  --vpn-connection-ids vpn-0123456789abcdef0

That helps narrow down whether a problem is tunnel establishment, BGP, routing, encryption parameters or the customer gateway configuration. Remember too that the 1.25 Gbps limit applies to a standard tunnel regardless of how much bandwidth the physical internet circuit has.

ExpressRoute

Azure ExpressRoute provides private connectivity between your network and Microsoft. The important point is that private doesn’t mean automatically encrypted. Microsoft offers MACsec encryption for ExpressRoute Direct, which encrypts traffic at Layer 2 between your network equipment and Microsoft’s edge routers, but it’s specific to ExpressRoute Direct rather than provider-provisioned circuits.

So instead of saying “ExpressRoute is private, so our traffic doesn’t need encryption”, the design question should be: what protection does this traffic need, and where is that protection provided? That’s a much better security discussion.

ExpressRoute private peering also has its own prefix limit. Microsoft accepts up to 4,000 IPv4 prefixes advertised from on-premises, or 10,000 with ExpressRoute Premium, which is far more generous than the Direct Connect limit covered below.

ExpressRoute MTU

ExpressRoute uses a maximum IP MTU of 1,500 bytes, as Microsoft documents in its ExpressRoute routing requirements. That matters when an on-premises network normally uses jumbo frames:

text

Server
  |
  | MTU 9000
  |
Switch
  |
Router
  |
ExpressRoute
  |
Azure

The local network supporting jumbo frames doesn’t mean the ExpressRoute path does. If an application starts behaving strangely after a migration, put MTU on the troubleshooting list.

AWS Direct Connect

AWS Direct Connect provides a private connection between your network and an AWS Direct Connect location. It comes in two connection types, and it’s worth knowing which one you’re buying:

TypeHow you get itSpeeds
DedicatedOrdered directly from AWSFixed port speeds: 1, 10, 100 and 400 Gbps
HostedSupplied through an AWS Direct Connect PartnerFrom 50 Mbps up to 25 Gbps, depending on the partner and location

You’ll often see Direct Connect described as ranging “from 50 Mbps to 100 Gbps”. That mixes the two types together, and it isn’t a range you can actually order from a single product. The current speeds and options are in the AWS Direct Connect User Guide.

Direct Connect isn’t automatically encrypted

Like ExpressRoute, Direct Connect provides private connectivity, but that doesn’t make application traffic encrypted end to end. If the security requirement calls for encryption, use it over the private connection, including appropriate VPN designs. Base the security model on the sensitivity of the traffic, not just on whether packets cross the public internet.

Direct Connect BGP prefix limits

This limit catches out larger environments. According to the AWS Direct Connect quotas, a private or transit virtual interface accepts a maximum of 100 routes per BGP session from on-premises, for IPv4 and IPv6 each. If you advertise more than 100, the BGP session goes idle and drops. A public virtual interface allows 1,000 routes, and AWS states that can’t be increased.

That’s a far tighter limit than Azure’s, so aggregation matters even more here. Instead of advertising a long list of individual subnets such as 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24 and so on, advertise an aggregate such as 10.10.0.0/16 wherever your routing allows it. It keeps you well under the limit and makes the routing table much easier to understand.

Direct Connect MTU

Direct Connect private virtual interfaces support an MTU of 1,500 or 9,001, and transit virtual interfaces support 1,500 or 8,500. Jumbo frames have to be supported along the whole relevant path, and enabling them can affect the underlying connection. That’s very different from ExpressRoute’s fixed 1,500, so don’t assume that because your EC2 instances support jumbo frames, the whole hybrid path does.

BGP is where the design becomes interesting

BGP is useful because it lets the network react to changes instead of relying entirely on static routes, but it also introduces its own failure modes. At a minimum, document the on-premises and cloud ASNs, BGP peer IPs, advertised and received prefixes, local preference, AS path, MED, route filtering and failover preference.

Take this design:

text

                 Primary
On-prem -------- ExpressRoute -------- Azure
   |
   | Backup
   +---------- VPN ------------------- Azure

You might want Azure routes learned over ExpressRoute to be preferred over the same routes learned through the VPN, and that preference has to be explicit. Otherwise you can end up with asymmetric routing:

text

Request:
On-prem -> ExpressRoute -> Azure

Response:
Azure -> VPN -> On-prem

Some applications tolerate this, but stateful firewalls generally don’t. A firewall that sees return traffic without having seen the request will usually drop it, and the symptom looks like an intermittent application failure rather than a routing problem.

ASN conflicts

Don’t pick an ASN at random because a field asks for one. Azure reserves several ASNs for its own use: the private ASNs 65515 and 65517 to 65520, and the public ASNs 8074, 8075 and 12076. ExpressRoute uses AS 12076 on the Microsoft side for Azure private and Microsoft peering, and it strips its internal private ASN when re-advertising routes. Choose your enterprise ASN deliberately and document it before building the BGP sessions:

text

Organisation ASN: 65010

Azure:
Microsoft ASN: 12076
On-prem peer ASN: 65010

AWS:
AWS-side ASN: documented by the selected gateway/VIF
On-prem peer ASN: 65010

The AWS-side ASN depends on the connectivity design, so use the generated AWS configuration rather than assuming the same value applies to every service.

MTU is one of the first things to test

MTU problems are particularly nasty because basic connectivity can still work. You might be able to ping 10.100.10.10 while an application fails as soon as it sends larger packets. On Windows, test with the Don’t Fragment flag set:

cmd

ping -f -l 1472 10.100.10.10

The 1,472-byte payload plus the 8-byte ICMP header and the 20-byte IPv4 header makes a 1,500-byte packet. If that fails, reduce the payload, for example with ping -f -l 1400 10.100.10.10, and keep working down. On Linux:

bash

ping -M do -s 1472 10.100.10.10

The goal is to find the largest packet that crosses the path without fragmentation. For VPN connections, remember that IPsec adds overhead, so the usable MTU can be lower than 1,500, and TCP MSS adjustment may be needed depending on the VPN implementation. This is one reason an application can work perfectly on the LAN but behave badly across a hybrid connection.

Don’t forget firewall policy

A hybrid connection isn’t a reason to allow everything between the networks. Avoid rules like this:

text

On-premises -> Azure
Source: Any
Destination: Any
Service: Any
Action: Allow

Instead, define the actual communication requirements:

text

On-prem application subnet
        |
        | TCP 443
        v
Azure application subnet

Azure backup subnet
        |
        | TCP 443 / required backup ports
        v
On-prem backup infrastructure

The routing layer decides where traffic can go, and the firewall decides what traffic is allowed to go there. They’re different controls.

Network segmentation matters more after the connection works

A common mistake is to treat the hybrid connection as one big extension of the LAN. Think about trust zones instead:

text

                 On-premises
                     |
                 Firewall
                     |
              Hybrid transit
                /          \
               /            \
          Azure              AWS
          /   \              /   \
     Web VNet App VNet    Web VPC App VPC

A compromised cloud workload shouldn’t automatically have a route to every on-premises server, and an infected workstation shouldn’t automatically reach every workload in Azure and AWS. Use separate subnets, route filtering, firewalls, security groups, NSGs, network ACLs where appropriate, private endpoints and application-level controls. The hybrid connection should provide the paths that are required, not become one giant trusted pipe.

Monitoring the connection

A green VPN tunnel doesn’t mean the application path is healthy, so monitor at several layers.

For the VPN, watch tunnel state, IKE status, IPsec security associations, packet counters, rekey events and tunnel flaps. For BGP, watch session state, prefixes received and advertised, route changes and session resets. On Cisco IOS:

text

show ip bgp summary
show ip route
show ip bgp

On a FortiGate:

text

get router info bgp summary
get router info routing-table all

In Azure, VPN diagnostics can help identify BGP disconnects and route problems, and Microsoft’s troubleshooting guidance covers checking BGP prefix counts and gateway route propagation. You can inspect a connection with:

bash

az network vpn-connection show \
  --resource-group rg-network \
  --name connection-onpremises

In AWS, use aws ec2 describe-vpn-connections for VPNs, and for Direct Connect:

bash

aws directconnect describe-virtual-interfaces
aws directconnect describe-direct-connect-gateway-attachments

These commands won’t replace packet captures or router diagnostics, but they help establish whether the cloud service believes the connection and routing objects are healthy.

Cost is part of the architecture

VPN isn’t automatically the right answer just because it’s cheaper. ExpressRoute and Direct Connect bring extra costs that can include port or circuit charges, connectivity-provider charges, cross-region traffic, data transfer, cloud gateway charges, VPN backup costs and redundant circuits. The exact price depends on region, provider, capacity and traffic pattern, so base the decision on your actual traffic requirements rather than a generic “Direct Connect is expensive” assumption.

For many organisations, the more useful calculation isn’t “how much does the circuit cost?” It’s “what does an hour of application downtime, degraded performance or a failed migration cost?” That can change the architecture decision considerably.

A practical production design

For a production environment with meaningful on-premises dependencies, a reasonable starting architecture is:

text

                        On-premises
                             |
             +---------------+---------------+
             |                               |
      ExpressRoute                    Direct Connect
       (primary)                         (primary)
             |                               |
             |         VPN backup paths      |
             |     +---------------------+   |
             |     |                     |   |
             v     v                     v   v
         Azure VNet                    AWS VPC

The exact topology depends on whether Azure and AWS need to talk to each other, whether you’re using a central transit architecture, and whether the on-premises network already has redundant WAN connectivity. Whatever the shape, aim for no overlapping address space, controlled route advertisements, BGP where dynamic routing helps, redundant paths with explicit primary and backup preferences, firewall inspection, segmentation, documented MTU, monitoring and tested failover.

Test the failure, not just the installation

A hybrid connection isn’t finished when the screen shows VPN: UP and BGP: ESTABLISHED. Test what happens when:

  • the primary or secondary VPN tunnel fails
  • a BGP session drops
  • the primary circuit fails
  • the firewall fails over
  • a route disappears, or one is advertised by mistake
  • the MTU is reduced
  • one cloud subnet becomes unreachable
  • the on-premises WAN fails

During each test, measure failover time, packet loss, route convergence, application recovery, firewall state and whether monitoring alerts fire. A connection that fails over in theory but causes a five-minute application outage may not meet the business requirement.

Build the network around failure

The technology is only one part of hybrid connectivity. A VPN can establish a secure tunnel in a relatively short time, and ExpressRoute and Direct Connect can provide private connectivity for larger production environments. But the connection itself doesn’t solve overlapping networks, route leaks, MTU problems, BGP limits or poorly designed firewall rules. Those are what decide whether the hybrid network is actually supportable.

Start with the address plan, then design the routes, then decide how the traffic should be protected and inspected, then build redundancy. Finally, test the failure scenarios before any application depends on the connection. That’s the difference between having a cloud connection and having a hybrid network you can actually operate.

Hybrid connectivity checklist

Before putting a production hybrid connection into service, check that:

  1. Azure and AWS address spaces don’t overlap with on-premises, and cloud subnet allocations are documented.
  2. The Azure GatewaySubnet is named correctly and is a /27 or larger.
  3. BGP ASNs are documented, valid and don’t use Azure’s reserved ranges.
  4. Only the required prefixes are advertised, aggregated well within the Azure and Direct Connect limits.
  5. BGP primary and backup preferences are explicit, and failover behaviour has been tested.
  6. MTU has been tested end to end, including any IPsec overhead.
  7. Firewall rules are restricted to the traffic actually required, and cloud workloads are segmented.
  8. VPN tunnels or private circuits have redundancy.
  9. Monitoring covers tunnels, BGP and traffic, and route changes generate alerts.
  10. Backup connectivity has been tested under failure.
  11. Cloud egress and connectivity costs are understood.
  12. The failure procedure is documented.

References

Leave a Reply

Your email address will not be published. Required fields are marked *