Skip to content

Go开启UDP SO_REUSEPORT支持

Linux的bind REUSEPORT选项是为了充分利用多核CPU,允许不同进程绑定同一个地址+端口号,避免出现仅仅有一个核在处理数据包,其他核闲着的问题。

启用前后的对比图(借用nginx的
reuseport before after

不过这么好的特性,Go默认是不开启的,需要自己手动调用net.ListenConfig(Go1.11以上)

一看API就懵了,咋是一个interface……

type ListenConfig struct {
// If Control is not nil, it is called after creating the network
// connection but before binding it to the operating system.
//
// Network and address parameters passed to Control method are not
// necessarily the ones passed to Listen. For example, passing "tcp" to
// Listen will cause the Control function to be called with "tcp4" or "tcp6".
Control func(network, address string, c syscall.RawConn) error
}

这是因为Go有近乎强迫症的向下兼容|backward compatible,不会在已有的接口上添加选项,所以肯定是新开一个API了。但是这个API跟常见POSIX API差太远了,仔细阅读了文档才摸索出下面这个例子:

import (
"net"
"syscall"
"golang.org/x/sys/unix"
)
func newUDPListener(address string) (conn *net.UDPConn, err error) {
// 构造Control闭包
cfgFn := func(network, address string, conn syscall.RawConn) (err error) {
// 构造fd控制函数
fn := func(fd uintptr) {
// 设置REUSEPORT
err = syscall.SetsockoptInt(int(fd),
syscall.SOL_SOCKET,
unix.SO_REUSEPORT, 1)
if err != nil {
return
}
}
if err = conn.Control(fn); err != nil {
return
}
return
}
lc := net.ListenConfig{Control: cfgFn}
lp, err := lc.ListenPacket(context.Background(), "udp", address)
if err != nil {
return
}
conn = lp.(*net.UDPConn)
return
}

效果方面,我的GoNTPD项目一到高峰期就没办法响应太多的请求,导致UDP recv buf 过高,图中的绿线是网卡吞吐量,红线就是UDP recv buf的使用量,越高就说明无法立即处理的请求越多,可以看到更新后,就没有出现UDP recv buf堆积的情况了。

UDP recv buf VS throughput

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.