Wirebus: Zero-Config, Re-Assignable IP Leases Anywhere

Sometimes, you just want to host something. Off your laptop, from your living room, inside a CI job. But you can’t, because nothing on the internet can reach you.

To state the obvious, there are not enough IP addresses. The framers of the internet imagined a network of peers intercommunicating, but provisioned only 32 bits to represent an IP address, enforcing a hard cap of about four billion IPs. This issue brought us IPv6, which allocates 128 bits to an IP address, novemseptuagintaoctillionupling the address space compared to IPv4. But it was built as a completely separate network: a packet can’t have an IPv4 source address and an IPv6 destination address.

Almost three decades on, enough of the internet is still IPv4-only that being reachable means being reachable over IPv4. It’s easy to rent a server with an IP address attached, but what if you already have a server, and you just want to get an IP to it?

Wirebus gets you one. You lease a public IP address, and every packet sent to it is delivered to any machine that can run WireGuard.

Wirebus isn’t a proxy. Most existing solutions focus on HTTP forwarding, but Wirebus works at the layer 3 level. You can use HTTP, TCP, UDP, GRE, IPsec, or any other crazy thing you can stick in a packet.

Wirebus is drop-dead simple. No custom client, no configuration. Wirebus lets you remap IPs without touching your machine, using a clever address rewriting mechanism.

Here’s how it works.

How Wirebus Works

Your machine connects to a Wirebus gateway over WireGuard and is handed 256 private IPv6 addresses. Each of these addresses acts as a “slot” to which a public IPv4 address may be assigned.

When a packet arrives at your leased public IPv4 address, the gateway rewrites it as an IPv6 packet with the same payload, and sends it down the tunnel to the IPv6 slot on your machine to which the IPv4 address is assigned. When your application replies, the reverse takes place: the IPv4 destination is read out of the IPv6 packet’s destination address. This means nothing has to be “remembered” between packets: everything the gateway needs to route the packet is contained within the packet itself.

Like other IPv4-to-IPv6 rewriting solutions, Wirebus defines a prefix wherein IPv4 addresses are to be embedded. When your app receives a packet, the original IPv4 source address will be found embedded in the IPv6 source address.

Critically, the prefix is configured to be routed out through the WireGuard tunnel. It’s because of this that no further configuration (connection marking and so on) is required to route reply packets correctly. The static WireGuard configuration supplies the translation prefix in AllowedIPs, so replies to incoming Wirebus connections are unambiguously destined for the Wirebus gateway.

That’s the whole mechanism. What may be less obvious is why this architecture is necessary to accomplish the goals. For the curious, let’s look at each existing approach and see where they fall short.

Goals & Prior Art

In designing Wirebus, we wanted an IP forwarding solution that would accomplish these goals:

1. Full layer 3 forwarding: the entire IP packet delivered to your machine (and back).
2. The ability to know where the packet came from.
3. Statelessness: for some of these protocols we might not be able to manage connections statefully in any smart way.
4. Use of existing standards so we have broad support and as little custom code as possible.
5. No custom agent on the system.
6. Works on any platform that supports a tunnelling mechanism.
7. IPs are movable between machines without touching either machine.

Layer 7 Solutions

Most existing solutions today exist at layer 7 of the OSI model. This is the highest level of the stack, where protocols like HTTP live. In fact, most solutions focus on supporting HTTP only.

There are some benefits to this approach:

• Most modern internet traffic is HTTP
• HTTP reverse-proxy infrastructure (such as the X-Forwarded-For header) is well supported
• HTTP can already be multiplexed across a single IP (using SNI and the Host header), which avoids dedicating an entire IP address to each endpoint

But there are considerable drawbacks as well:

• Server machines need to have a custom agent from the provider installed
• Of course, the provider must offer support for every application protocol you want to support
• In practice, providers generally support only HTTP

For casual work on HTTP-based applications and APIs, this solution is sufficient. However, it’s not really an IP-forwarding solution: your application sits behind a CDN. You’re renting reachability, not an IP address. In terms of our stated goals, this fails 1, 3, and 5. For more complex applications, we need something else.

Layer 4 Solutions

Many providers of layer 7 solutions also offer a layer 4 solution. Rather than forwarding traffic at the HTTP level, the provider’s edge accepts TCP connections and relays them to an agent on your machine. The agent opens a matching connection to your service and relays the data in the TCP stream.

Since the agent opens the connection to your service, your service sees the agent as the client. To recover the true source address, the agent and your service can use the PROXY protocol, which embeds the source IP in the TCP stream. Both sides must agree beforehand to use the protocol, and your service needs to be able to decode the source IP.

Typically, such services offer TCP only. Some also support UDP, but there’s no existing support for other protocols like GRE, IPIP, IPsec, or SCTP. Like the layer 7 solutions, you also need the provider’s agent on your machine. This fails 1, 3, 5, and usually 2.

Bespoke Tunnelling Solutions

The closest prior art in this category is something hobbyists often do. You can spin up a VPS somewhere and get a public IP. Then you can run a VPN to the VPS, and the VPS forwards traffic over the VPN to your machine.

There is a subtle issue with this solution: the reply packets.

At home, port forwarding works effortlessly because the router that’s intermediating the incoming connection is the same as the default route. Any internet-bound packet is destined for the same router, regardless of where it came from.

With a VPS in the picture, you now have two paths in and two paths out. Ordinary destination routing will send any packet destined for the internet to your default route, whether it’s a reply to an incoming connection or part of an outgoing connection. Replies to incoming connections, if sent via the default route, carry the wrong source address and are killed (if not by your router, then by BCP 38).

Of course, you can set the default route to the VPS, but at that point you’ve built a VPN, not an IP lease: all traffic in and out goes through the VPS. Now all your web browsing has a data centre IP attached; you’re about to get really good at filling out CAPTCHAs.

For this to work properly, your machine has to route replies based on where the request came in, rather than where the reply is going. There are three ways to do this:

More restrictive platforms like iOS support none of these. On macOS, the standard networking APIs don’t support this, so you have to deal with the pf subsystem directly, which could cause trouble with other firewalls and VPNs on the system.

Then there’s the VPS itself: with this solution, you have to pay for an entire VPS, and either continue to pay for it when you’re not using it, or build your own tooling to automate setup and teardown.

On paper, the VPN-to-VPS solution meets most of our stated goals. In practice, getting it working is time-consuming and costs you a server. Even then, it fails goal 6. As for goals 3 and 7, you can have one or the other, but not both. You can have the VPS NAT the public IP to a private IP assigned to your machine, making the IPs remappable. But NAT requires conntrack, so the forwarding is stateful (and only supports protocols conntrack undestands). If instead the VPS routes the public IP down the tunnel and your machine binds it directly, forwarding is stateless, but moving this IP from one machine to another now requires reconfiguring both machines.

Note what the three fixes have in common: each routes replies by something other than their destination. Operating systems vary in their level of implementation for this method of routing. Routing by destination, in contrast, is the same everywhere. By making the reply’s destination field the deciding factor on where the packet is routed, Wirebus lets you operate on any platform that WireGuard runs on, including the most restrictive ones like iOS.

SIIT-DC

We didn’t invent the translation mechanism described above. The address mapping table and translation prefix are codified as SIIT-DC. This architecture was designed for a data centre that wants to open IPv6-only servers to the IPv4 world:

• You have a network of servers that are IPv6 only.
• At the border of this network you have a dual-stack server: it’s connected to the IPv6 server network, and it has some globally routable IPv4 addresses as well.
• The border server runs IPv6/IPv4 translation, and it has an EAMT (Explicit Address Mapping Table) which defines how to map IPv4 and IPv6 addresses.

The Wirebus “slot table” is the SIIT-DC EAMT. What’s different is where the IPv6 side lives. SIIT-DC assumes the servers are on a data centre LAN, each with an IPv6 address. Wirebus instead has the servers living at the end of a WireGuard tunnel, with each owning 256 addresses. In this way, the tunnel config is written once, and any change to the address mapping happens on the gateway side.

In other words, from the perspective of the core networking stack, Wirebus is the SIIT-DC architecture applied to a system where the IPv6-only servers are distant endpoints connected to the gateway via WireGuard.

Forwarding IPv6 Traffic

A sharp-eyed reader will notice an issue. What if we want to apply this technique to IPv6? That is, we want to be able to forward IPv6 connections to your machine, and remap those addresses at will, without any configuration change. The magic here lies in the embedding of an IPv4 address in an IPv6 prefix, and you can’t embed IPv6 in IPv6.

… right?

Well, it turns out the IPv6 address space is very big. If you map out the 16 blocks of IPv6 by leading nibble, all of global unicast lives in just two of them, and most of these blocks are reserved for future use.

So if we want to embed a global unicast IPv6 address in another unicast IPv6 address, can we just flip the three bits in the front like this?

Absolutely. At least until IANA allocates c000::/3.

Concretely: the gateway flips the client’s source address into the c000::/3 space on the way in. Your machine’s WireGuard config carries this space in AllowedIPs, so the replies go into the tunnel, and the gateway flips them back into 2000::/3 before sending them along to the client.

Here is the whole journey for both families, hop by hop. Each message is one packet, written as source → destination:

Limitations

While we think Wirebus is a better solution to IP leasing than the prior art in this category, there are some limitations to be aware of on both the provider and customer side.

First, your server application has to support IPv6. Of course, the client does not – it connects to your server over IPv4, but your server receives these packets as translated IPv6. This is mainly a problem for multiplayer game servers, many of which do not support IPv6 at all.

Second, there’s some MTU cost. Because an IPv6 header is bigger than an IPv4 header, you lose 20 bytes of MTU on the conversion. You lose some again when you embed that IPv6 packet in WireGuard. Luckily, ICMP works with this solution, so PMTUD should also work. It doesn’t always, but when it doesn’t, clamping TCP MSS is always an option.

Third, the potential for abuse means unvetted customers must not be able to initiate outgoing connections. Since Wirebus is stateless by design, clever and malicious customers could make outbound connections, faithfully translated by the gateway. This effectively makes Wirebus into a consumer VPN, with all the abuse potential and IP reputation pitfalls that entails.

Currently, we solve this problem by putting a stateful firewall in front of unvetted customers, like so:

Sadly, this means that most customers do not get support for certain exotic protocols we worked so hard to enable. To be clear, most protocols other than TCP and UDP do continue to work for incoming connections, but some protocols with a strong commitment to statelessness may choke if the firewall believes that the customer side is opening a connection to the outside world, rather than the other way around.

In the future, we could enable statelessness for all customers by doing some more careful filtering, such as looking for outgoing TCP SYN packets and suspicious UDP traffic. For now, we require customers to contact us to explain their use case, and we flip a config bit to turn their stateful firewall off.

Implementing Wirebus

Much of what we need is built into the Linux kernel. WireGuard was upstreamed in 2020, and we need only the userspace companion tools to interact with it.

For IPv4, we have one out-of-tree kernel module, the Jool project, which implements many IPv6 transition mechanisms, including SIIT-DC.

For IPv6 there is no Jool. A leased IPv6 address reaches its slot through an nft map, the IPv6 counterpart of the EAMT. One map sends the leased address to the slot on the way in, and its mirror puts the leased address back on the way out:

map transplant_in  { type ipv6_addr : ipv6_addr }   # leased address : slot
map transplant_out { type ipv6_addr : ipv6_addr }   # slot : leased address

ip6 daddr set ip6 daddr map @transplant_in    # inbound: leased address -> slot
ip6 saddr set ip6 saddr map @transplant_out   # reply:   slot -> leased address

The client’s own address is carried across by flipping its top three bits, 2000::/3 <-> c000::/3. That needs only one more rule, and because the flip is its own inverse, the same XOR on the destination undoes it on the reply:

ip6 saddr set ip6 saddr ^ e000::

Neither rewrite keeps a table of flows. The gateway looks at each packet on its own, exactly as Jool does for IPv4.

The one thing nft cannot express is the matching rewrite inside ICMPv6 error messages, since that write must update the ICMPv6 checksum. Those rules are handed to the kernel through pyroute2.

expr("payload", dreg=1, base=TRANSPORT_HEADER, offset=16, len=16)   # load the embedded source address
expr("bitwise", sreg=1, dreg=1, len=16, mask=ALL_ONES, xor=FLIP)    # flip its top three bits
expr("payload", sreg=1, base=TRANSPORT_HEADER, offset=16, len=16,
     csum_flags=["NFT_PAYLOAD_L4CSUM_PSEUDOHDR"])                   # write it back, patching the ICMPv6 checksum

Note that this is a one-time setup, not a daemon that processes packets in userspace.

We implemented a control plane and web UI that issues commands to the data plane according to user configuration. Its main purpose is to adjust the EAMT to route leased IPs to where users assign them. Naturally, it also manages business logic such as billing, auth, and account suspension. The control and data planes are independent: if the control plane breaks, traffic continues to flow as usual (of course, the configuration can’t be altered until the control plane comes back up).

On the customer side, each machine gets a one-time WireGuard config like this one:

[Interface]
PrivateKey = <your private key>
Address = fded:da7a:ba5e:beef::1a00/128
Address = fded:da7a:ba5e:beef::1a01/128
Address = fded:da7a:ba5e:beef::1a02/128
# ... one line per slot, 256 in all ...
Address = fded:da7a:ba5e:beef::1aff/128

[Peer]
PublicKey = <gateway public key>
Endpoint = <wirebus gateway server>
AllowedIPs = <private wirebus space>, c000::/3
PersistentKeepalive = 25

Once set, it never changes, unless the customer wants to rotate their private key, or move to a different gateway server.

Where That Leaves Us

You have a server, and you want an IP to it.

Existing solutions handle common cases, but they come with trade-offs that we weren’t happy with. We identified these trade-offs and built Wirebus to resolve all of them with as light a touch as possible.

What you get is a public IP that delivers whole packets, sender’s address intact, to anything that runs WireGuard, and moves between machines without either of them noticing. What it costs you is an application that speaks IPv6 and a few bytes of MTU. We think that’s a good trade.