Guides
Layer-4 forwarding
TCP / UDP port forwarding: port pools, L4 apps, PROXY protocol, DNS, statistics, and node ports.
Concepts
| Term | Definition |
|---|---|
| L4 app | Forwards TCP connections or UDP sessions on one port of every node of a cluster to its origins. |
| Port pool | A range of ports the cluster's L4 apps may use, per protocol (TCP, UDP, TCP + UDP). |
| Reserved port | A port of the cluster's HTTP / HTTPS listeners; never part of a port pool. |
| UDP session | The datagrams from one client address and port; one session until the idle timeout. |
| PROXY protocol | A header at the start of a connection that carries the client's address and port; v1 is text, v2 binary. |
How it works
- Port ranges are set on the cluster's Port pools tab; an L4 app's port must fall inside one.
- Saving an L4 app publishes the cluster's configuration revision (through the configuration canary as for sites; disabling and deleting reach every node at once); every node of the cluster listens on the port and forwards connections to the origins by weight.
- With DNS steering on for the cluster, every enabled app has the CNAME target
<app UUID>.<cluster domain>, through which clients reach healthy nodes. - Nodes report per app and per minute the connections, refusals, peak concurrency, and bytes, shown on the app's Analytics tab.
Nodes need the capability l4-v1, see Node requirements.
Set up port pools
- Open Clusters & nodes, select the cluster, and switch to the Port pools tab (
/clusters?tab=ports, shown once the cluster has a node). - Click Add port pool, select the Protocol (TCP, UDP, or TCP + UDP), and fill in First port and Last port. For a single port both are the same.
- Click Save. The console shows Saved.
- Open these ports per protocol in the node hosts' firewall and the cloud security group; container nodes also publish them, see Add nodes.
Reserved ports at the top of the card lists the ports of the cluster's HTTP / HTTPS listeners; L4 apps next to the title opens the cluster's L4 app list.
| Item | Behavior |
|---|---|
| Ports | 1024–65535; the first port must not be above the last ("The first port must not be above the last") |
| Count | At most 64 port pools per cluster |
| Overlap | Pools of one protocol must not overlap, and TCP + UDP overlaps both TCP and UDP ("Port pools overlap: …"); the conflicting pools are marked |
| Shrinking | Refused when the pools no longer contain the port of an app, disabled apps included ("Ports in use by …") |
| Publishing | Port pools only check app ports: saving publishes no configuration revision, nodes are not affected |
| Audit | cluster.port_pools_update, with the pools before and after as metadata |
Create an L4 app
-
Open Sites, switch to L4 apps with the Sites | L4 apps switch above the list (
/l4), and click New L4 app. Searching L4 apps with ⌘K / Ctrl+K opens the page too. -
Fill in Name (at most 100 characters); with several clusters, select the Cluster.
-
Select the Protocol (TCP or UDP) and fill in Listen port; for a range of ports also Last port. The hint Port pools: … below lists the pools of that protocol; when the cluster has none, the hint reads "The cluster has no TCP port pool" and Set up port pools opens the cluster's Port pools tab.
-
Under Origins, fill in Origin, Port, and Weight, and turn on Backup where needed; Add origin adds more, see Origins and health checks.
-
Set PROXY protocol, Timeouts, Passive health check, IP lists, and Limits per node as needed.
-
Enabled is on by default. Click Create; the console shows "Created, revision #N".
-
Verify: the app appears in the list and CNAME target shows
<app UUID>.<cluster domain>; connect to the port on any node:nc -vz <node IP> 9000 # TCP: the connection succeeds dig @<node IP> -p 5353 example.com +short # UDP, here forwarding to a DNS origin -
The app's Analytics tab starts showing connections (reported every minute, after about 1–2 minutes).
List and app page
The L4 apps list is sorted by port, with the columns Name, Protocol, Listen port, Origins, CNAME target, and Enabled; with several clusters, select All clusters or one cluster. Row menu: Edit, Analytics, Delete.
Click a name to open the app page (/l4/<app ID>): the Overview tab shows the status, listen port, cluster, CNAME target (with the target of every line), origins, PROXY protocol, timeouts, passive health check, IP lists, limits per node, and update time, with Edit and Delete; the Analytics tab is described in Statistics.
| Action | Where | Behavior |
|---|---|---|
| Edit | Row menu or Edit on the app page, dialog Edit L4 app | Same fields as for creating; the cluster cannot change. Saving shows "Saved, revision #N" |
| Disable / enable | The Enabled switch in the list, or the switch next to Status on the app page; asks first, and disabling notes "Nodes stop listening on the port and its DNS record is removed" | A disabled app keeps its port and settings, is not shipped to nodes, and has no DNS record; setting the current state publishes nothing |
| Delete | Row menu or Delete on the app page, confirm "Delete L4 app {name}?" | Deletes the app, its origins, and its statistics |
| Change | Revision reason | Audit |
|---|---|---|
| Create | L4 application {app} created | l4_app.create |
| Edit | L4 application {app} updated | l4_app.update (changed fields with before and after) |
| Disable / enable | L4 application {app} updated | l4_app.disable / l4_app.enable |
| Delete | L4 application {app} deleted | l4_app.delete |
Origins and health checks
| Field | Values | Default | Effect |
|---|---|---|---|
| Origin | Host name or IP | — | Follows the origin address restrictions: special-purpose addresses only inside the origin allow list; nodes check every address a host name resolves to |
| Port | 1–65535 | — | Origin port; may differ from the listen port |
| Weight | 1–100 | 1 | Random choice by weight |
| Backup | On / off | Off | Used only while every primary origin is down |
An app has 1–32 origins, at least one of them not a backup ("Keep at least one origin that is not a backup").
| Item | Behavior |
|---|---|
| Choice | Healthy primary origins at random by weight; when every primary is down, only healthy backups; when all are down, all of them in turn, primaries first |
| Retries | The next origin only after a failed connection to the origin (refused, timed out), at most 3 origins per connection; no switch once connected |
| Host names | Resolved by the resolver nodes use for sites; a failed resolution, or only refused addresses, counts as one failure |
Passive health check and timeouts:
| Field | Values | Default | Effect |
|---|---|---|---|
| Passive health check · Failures before down | 1–100 | 3 | After this many failed connections (for UDP also an unreachable origin, such as ICMP port unreachable) the origin is taken out |
| Passive health check · Retry after (seconds) | 1–3600 | 30 | How long the origin stays out before it is tried again; also how long failures count, one success clears them |
| Timeouts · Connect (seconds) | 0.1–60 | 5 | Timeout for connecting to an origin |
| Timeouts · Idle (seconds) | 1–86400 | TCP 600, UDP 30 | How long without data in either direction ends a connection or session; switching the protocol in the dialog sets the protocol's default (unless edited) |
Health state lives on each node only: it is not reported to the console and not shown in the UI. On the node host, list the origins currently down:
sudo curl -s --unix-socket /run/edgeweir-node/control.sock http://localhost/v1/l4down in the response lists {app_id, origin_id, down_until}.
Port ranges and origin ports
| Field | Values | Default | Effect |
|---|---|---|---|
| Last port | Empty, or above the listen port; at most 1000 ports per app | Empty | The app listens on every port from Listen port to Last port; the whole range must be inside the protocol's port pools (adjacent pools may share it) and must not overlap another app's port or range |
| Origin port | Fixed / Same as the port | Fixed | Fixed: each origin's own port; same: the origin port is the port the connection arrived on (9005 of range 9000-9010 connects to the origin's 9005), the origins' port fields stay empty |
| Item | Rule |
|---|---|
| Cluster limit | A cluster's L4 apps listen on at most 2048 ports together (ranges count each port; "A cluster's L4 applications use at most 2048 ports") |
| Node resources | A TCP range takes one listening socket per port; UDP one per port and worker, see Ports and firewalls |
| Statistics and DNS | Per app, as for a single port |
| Node capability | Ranges, origins on the arriving port and TLS termination need l4-v2 |
TLS termination
A TCP app may choose a TLS certificate: the node terminates TLS and connects to the origins over plain TCP.
| Field | Values | Default | Effect |
|---|---|---|---|
| TLS certificate | No TLS / an issued, unexpired certificate nodes can load (certificates listed as Unusable cannot be chosen; see troubleshooting in HTTPS) | No TLS | The certificate of the handshake |
| Minimum TLS version | TLS 1.2 / TLS 1.3 | TLS 1.2 | Handshakes below it are refused |
| Item | Behavior |
|---|---|
| SNI | Must be one of the certificate's names (a wildcard covers one label), else the handshake is aborted; a handshake without SNI gets the certificate |
| Cipher suites | The sites' Modern profile, not configurable |
| With the PROXY protocol | Both work together: with Accept PROXY protocol the header comes before the handshake; Send to origins still sends one |
| IP lists and limits | Checked after the handshake |
| Certificate | A certificate an app uses cannot be deleted ("The certificate is still used by L4 applications: …"); after a renewal nodes use the new one without a reload |
| UDP | Not supported ("TLS termination is for TCP applications only") |
Verify:
echo NAME | openssl s_client -quiet -connect <node IP>:9000 -servername game.example.comPROXY protocol
TCP apps only; UDP apps show "UDP does not carry PROXY protocol".
| Field | Values | Default | Effect |
|---|---|---|---|
| Send to origins | None / v1 / v2 | None | Every connection to an origin starts with a PROXY protocol header: the client's address and port, and the node address and port it connected to. v2 headers carry no TLVs |
| Accept PROXY protocol | On / off | Off | The listener expects every connection to start with a PROXY protocol header, v1 or v2 detected automatically; for nodes behind a layer-4 load balancer. The client address is the one in the header (the TCP peer for UNKNOWN), and IP lists check that address |
| Accept | Send to origins | Forwarding |
|---|---|---|
| Off | Any | Native nginx forwarding |
| On | None | Native nginx forwarding |
| On | v1 / v2 | Lua relay: the header it writes carries the client address from the received header (a header nginx sends natively would carry the load balancer's address). Lower throughput than native forwarding; no half-close, one side closing ends the connection |
With Accept PROXY protocol on, connections without a header are closed, and a client that reaches the port directly can claim any address in the header: open the port to the load balancer only. Both settings are structural like the port and protocol: changing them reloads the nodes, see Reloads and long connections.
IP lists and connection limits
| Field | Values | Default | Effect |
|---|---|---|---|
| IP lists · Allow lists | Any IP lists, at most 16 | None | When set, only addresses in these lists are accepted |
| IP lists · Block lists | Any IP lists, at most 16 | None | Addresses in these lists are refused |
| Limits per node · Concurrent connections | 0–10000000 | 0 | Connections (UDP: sessions) a node forwards at once; 0 means no limit |
| Limits per node · New connections per second | 0–1000000 | 0 | New connections (UDP: new sessions) a node accepts per second; 0 means no limit |
Add list picks from IP lists; a list's Action does not matter here.
| Item | Behavior |
|---|---|
| Order | Allow lists, block lists, new connections per second, concurrent connections; the first one not met refuses |
| Refusal | Nothing is forwarded: TCP connections are closed (data the client sent and nobody read makes the kernel answer RST), UDP datagrams are dropped; counted as Refused. A refused UDP client counts once per datagram |
| Counting | Limits count per node; the cluster's total capacity is the number of nodes × the limit |
| List changes | Nodes apply changes to list entries without a reload; a list referenced by an L4 app cannot be deleted ("The IP list is used by …", naming the users) |
| Other protection | Global Block and Allow lists, site bans and HTTP-layer bans, rules, WAF, challenges, and CC protection do not apply to L4 apps. On nodes with kernel-ban-v1, global bans drop packets in the kernel with nftables, L4 ports included, see Kernel bans |
DNS
L4 apps use the cluster's DNS steering like sites, see DNS steering and alerts.
| Item | Behavior |
|---|---|
| Records | One <app UUID>.<cluster domain> CNAME all.<cluster domain> per enabled app; with Keep per-site line targets also <line name>.<app UUID>.<cluster domain> |
| Steering | Resolution lines, backup node groups, health removal, and scheduling rules apply as usual |
| Health | Nodes are removed by node health (heartbeat, data plane, current revision applied, scheduling rules), not by the origin state of a single app |
| Writing | After creating or enabling, the next reconciliation (every minute) writes the record; manual mode lists it among the records to create by hand |
| Disabled | A disabled app has no record; the next reconciliation deletes it (disabled sites keep theirs) |
| DNS not managed | CNAME target shows "Cluster DNS is off" |
Clients connect to the CNAME target with the app's port, for example on your own domain:
game.example.com. CNAME <app UUID>.<cluster domain>.Clients connect to game.example.com:9000. CNAME target on the app page also lists the target of every line, for clients that should use one line only.
Statistics
The Analytics tab of the app page (/l4/<app ID>?tab=stats), with the ranges Last hour, Last 6 hours, Last 24 hours (default), and Last 7 days; Refresh reads again.
| Metric | Meaning |
|---|---|
| Connections | TCP connections or UDP sessions accepted |
| Refused | Connections or sessions refused by the IP lists or the limits per node |
| Peak concurrency | Per minute the sum of the nodes' peaks, the highest minute of the range |
| Received | Bytes received from clients |
| Sent | Bytes sent to clients |
Charts Connections (connections and refused) and Traffic (received and sent); the Nodes table lists every reporting node's values over the range, busiest first.
| Item | Behavior |
|---|---|
| Reporting | Nodes report per app and per minute, in the same batch as site statistics; bytes of open connections are sampled every 10 seconds, so long connections count in every minute |
| Retention | Minute data for 7 days; deleted with the app |
| Loss | Deleting a cluster's last L4 app, or an nginx restart, loses the last minute or two a node has not reported yet |
| Node metrics | A node's active connections (metrics-v1) include layer-4 client connections, upstream connections, and UDP sessions, and so does the Active connections condition of scheduling rules |
Reloads and long connections
| Change | Node behavior |
|---|---|
| Port, last port, protocol, whether TLS is terminated, Accept PROXY protocol, the version under Send to origins; creating, deleting, disabling, enabling | Structural: nginx.conf is rendered again and nginx reloads |
| Origins, weights, backups, the origin port mode, passive health check, timeouts, IP lists and their entries, limits per node, the chosen certificate and minimum TLS version | Hot update without a reload; new settings apply to later connections |
| Item | Behavior |
|---|---|
| TCP connections | Connections opened before a reload stay with the old worker and keep forwarding until they end or reach the idle timeout; the old worker exits after its last connection |
| UDP sessions | Do not survive a reload: later datagrams start a new session in a new worker (possibly with another origin); the old session ends at the idle timeout |
| New ports | Connections that arrive between the reload and the push of the L4 app table (milliseconds) are closed; the node reports the configuration applied only after the push |
| Old workers | By default they serve their connections until these end, so frequent structural changes leave several sets of old workers. The node option --stream-shutdown-timeout makes old workers close the connections they still serve after that long, HTTP keep-alive and WebSocket connections included, see Add nodes |
UDP
| Item | Behavior |
|---|---|
| Sessions | One client address and port is one session until the idle timeout; any number of replies per datagram from the origin is forwarded (DNS, QUIC, game protocols work) |
| Idle timeout | 30 seconds by default. Set a short idle timeout for request-response protocols such as DNS, otherwise every client port holds a concurrent session until it times out |
| PROXY protocol | Not supported |
| Refusal | Refused datagrams are dropped and counted one by one |
Node requirements
| Item | Behavior |
|---|---|
| Capability | l4-v1 once the configuration has an enabled L4 app |
| Nodes without it | Refuse configurations with L4 apps, keep their last-known-good configuration, and show Upgrade required in Clusters & nodes; since they do not apply the current revision, they leave DNS steering after 2 minutes. The port pools tab, the app list, and the dialog warn "Nodes {nodes} of {cluster} lack L4 forwarding and refuse configurations with L4 apps until upgraded" |
| Order | Upgrade the nodes first, then create L4 apps |
| Ports | Open the port pools in the node hosts' firewall and the cloud security group, and publish them on container nodes, see Add nodes |
| Privileges | Ports are 1024 or higher: the node's systemd unit and image need no extra privileges |
Rollback
Roll back in the cluster's Revisions ships the L4 app settings of the chosen revision:
| Case | Behavior |
|---|---|
| The app is disabled now | Not shipped |
| The app was deleted since, its port is no longer inside a port pool, or a referenced IP list was deleted | The rollback is refused ("Rollback references resources that are no longer assigned or available") |
| The configuration canary rolls back to the stable revision | These apps are left out, everything else is shipped |
A rollback does not change the apps' saved settings; any later change publishes them again from the saved settings.
Limits
| Item | Limit |
|---|---|
| Apps | At most 256 per cluster ("A cluster can have at most 256 L4 applications"), disabled ones included |
| Port ranges | At most 1000 ports per app; 2048 ports per cluster |
| TLS termination | TCP only; one certificate per app |
| Ports | 1024–65535, inside a port pool of the cluster for the protocol; a port of a protocol belongs to one app of the cluster (disabled apps included) |
| Port pools | At most 64 per cluster |
| Origins | 1–32 per app, at least one not a backup |
| IP lists | At most 16 allow lists and 16 block lists |
| PROXY protocol | TCP only; v2 headers carry no TLVs; accepting and sending together uses the Lua relay |
| Passive health state | On the node only |
| Statistics | Per minute, kept for 7 days |
| Protection | Only the app's own IP lists, the limits per node, and kernel bans |
| API | Service accounts cannot call the port pool and L4 app procedures, see API and endpoints |
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| "Port {port} is outside the cluster's port pools for this protocol" | The port is in no pool of the protocol | Add a port pool on the cluster's Port pools tab, or use another port |
| "Ports in use by {apps}" | Creating or editing: another app of the cluster (disabled ones included) uses the port and protocol; saving port pools: an app's port would be left outside | Use another port, or change or delete the listed apps first |
| "Port pools overlap: {pools}" | Pools of one protocol overlap; TCP + UDP overlaps TCP and UDP | Change the ranges or protocols |
| "A cluster can have at most 256 L4 applications" | The limit is reached | Delete unused apps |
| "PROXY protocol is for TCP applications only" | The API set PROXY protocol on a UDP app | Remove the PROXY protocol settings |
| "Nodes {nodes} of {cluster} lack L4 forwarding …" | Nodes lack l4-v1 | Upgrade the nodes |
| Some nodes leave DNS after an app is created | As above: these nodes do not apply the current revision | Upgrade the nodes, or disable the app first |
| Connections time out | The node firewall or security group does not open the port; a container node does not publish it | Open it as in Ports and firewall |
| Connections are closed at once | Refused by an IP list or a limit (Refused rises); Accept PROXY protocol is on but the client sends no header; the port was just added | Check the lists and limits; turn on Accept PROXY protocol only behind a load balancer |
| The origin drops the connection or reports a protocol error right after connecting | Send to origins is set but the origin does not accept PROXY protocol, or expects another version | Enable that version on the origin, or select None |
| The origin sees the load balancer as the client | A layer-4 load balancer sits in front of the nodes | Have the load balancer send PROXY protocol, turn on Accept PROXY protocol, and choose a version under Send to origins |
| Every connection fails and Refused does not rise | No origin is reachable | Check down of GET /v1/l4 on a node, the origin addresses, and the origin's firewall |
| High UDP Peak concurrency, or the Concurrent connections limit reached | Every client port is a session until the idle timeout | Lower Idle (seconds) |
| Old nginx workers stay after a reload | They still serve long connections | Expected; set --stream-shutdown-timeout to bound it |
| CNAME target shows "Cluster DNS is off" | The cluster's DNS mode is Not managed | Bind the cluster on its DNS tab |
| "The IP list is used by …" | Deleting a list the listed apps still reference | Remove it from the app's allow and block lists first |
Regional probes and scheduling
Regional probes, node metrics, scheduling addresses with backup IP levels, and scheduling rules that change DNS by metrics.
Access logs and access keys
Traffic analytics, access log sampling, search, and export, analytics storage modes, and creating and revoking access keys.