dartnode

Remove caCertPool errCert “else” in p2pnetwork.go

.gitignore

@@ -15,4 +15,10 @@ libs/
 openp2p.app.jks
 openp2p.aar
 openp2p-sources.jar
-build.gradle
+wintun/
+wintun.dll
+.vscode/
+app/.idea/
+*_debug_bin*
+cmd/openp2p
+vendor/

README-ZH.md

@@ -19,9 +19,9 @@
 
 [查看详细](#安全性)
 ### 4. 轻量
-文件大小2MB+，运行内存2MB+；全部在应用层实现，没有虚拟网卡，没有内核程序
+文件大小2MB+，运行内存2MB+；它可以仅跑在应用层，或者配合wintun驱动使用组网功能
 ### 5. 跨平台
-因为轻量，所以很容易支持各个平台。支持主流的操作系统：Windows,Linux,MacOS；和主流的cpu架构：386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64
+因为轻量，所以很容易支持各个平台。支持主流的操作系统：Windows,Linux,MacOS；和主流的cpu架构：386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64、s390x、ppc64le
 ### 6. 高效
 P2P直连可以让你的设备跑满带宽。不论你的设备在任何网络环境，无论NAT1-4（Cone或Symmetric），UDP或TCP打洞,UPNP,IPv6都支持。依靠Quic协议优秀的拥塞算法，能在糟糕的网络环境获得高带宽低延时。
 
@@ -31,7 +31,7 @@ P2P直连可以让你的设备跑满带宽。不论你的设备在任何网络
 ## 快速入门
 仅需简单4步就能用起来。  
 下面是一个远程办公例子：在家里连入办公室Windows电脑。  
-（另外一个快速入门视频 https://www.bilibili.com/video/BV1Et4y1P7bF/）
+（另外一个快速入门视频 <https://www.bilibili.com/video/BV1Et4y1P7bF/>）
 ### 1.注册
 前往<https://console.openp2p.cn> 注册新用户，暂无需任何认证
 
@@ -96,7 +96,7 @@ Windows默认会阻止没有花钱买它家证书签名过的程序，选择“
 服务端有个调度模型，根据带宽、ping值、稳定性、服务时长，尽可能地使共享节点均匀地提供服务。连接共享节点使用TOTP密码，hmac-sha256算法校验，它是一次性密码，和我们平时使用的手机验证码或银行密码器一样的原理。
 
 ## 编译
-go version go1.18.1+  
+go version 1.20 only (支持win7) 
 cd到代码根目录，执行
 ```
 make
@@ -128,13 +128,15 @@ CGO_ENABLED=0 env GOOS=linux GOARCH=amd64 go build -o openp2p --ldflags '-s -w '
 6. 客户端提供WebUI
 7. ~~支持自有服务器，开源服务器程序~~(100%)
 8. 共享节点调度模型优化，对不同的运营商优化
-9. 方便二次开发，提供API和lib
+9. ~~方便二次开发，提供API和lib~~(100%)
 10. ~~应用层支持UDP协议，实现很简单，但UDP应用较少暂不急~~(100%)
-11. 底层通信支持KCP协议，目前仅支持Quic；KCP专门对延时优化，被游戏加速器广泛使用，可以牺牲一定的带宽降低延时
+11. ~~底层通信支持KCP协议，目前仅支持Quic；KCP专门对延时优化，被游戏加速器广泛使用，可以牺牲一定的带宽降低延时~~(100%)
 12. ~~支持Android系统，让旧手机焕发青春变成移动网关~~(100%)
-13. 支持Windows网上邻居共享文件
-14. 内网直连优化，用处不大，估计就用户测试时用到
+13. ~~支持Windows网上邻居共享文件~~(100%)
+14. ~~内网直连优化~~(100%)
 15. ~~支持UPNP~~(100%)
+16. ~~支持Android~~(100%)
+17. 支持IOS
 
 远期计划：
 1. 利用区块链技术去中心化，让共享设备的用户有收益，从而促进更多用户共享，达到正向闭环。

README.md

@@ -19,10 +19,10 @@ The code is open source, the P2P tunnel uses TLS1.3+AES double encryption, and t
 [details](#Safety)
 
 ### 4. Lightweight
-2MB+ filesize, 2MB+ memory. It runs at appllication layer, no vitrual NIC, no kernel driver.
+2MB+ filesize, 2MB+ memory. It could only runs at application layer, or uses wintun driver for SDWAN.
 
 ### 5. Cross-platform
-Benefit from lightweight, it easily supports most of major OS, like Windows, Linux, MacOS, also most of CPU architecture, like 386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64.
+Benefit from lightweight, it easily supports most of major OS, like Windows, Linux, MacOS, also most of CPU architecture, like 386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64、s390x、ppc64le.
 
 ### 6. Efficient
 P2P direct connection lets your devices make good use of bandwidth.  Your device can be connected in any network environments, even supports NAT1-4 (Cone or Symmetric),UDP or TCP punching,UPNP,IPv6.  Relying on the excellent congestion algorithm of the Quic protocol, high bandwidth and low latency can be obtained in a bad network environment.
@@ -103,7 +103,7 @@ That's right, the relay node is naturally an man-in-middle, so AES encryption is
 The server side has a scheduling model, which calculate  bandwith, ping value,stability and service duration to provide a well-proportioned service to every share node. It uses TOTP(Time-based One-time Password) with hmac-sha256 algorithem, its theory as same as the cellphone validation code or bank cipher coder.
 
 ## Build
-go version go1.18.1+  
+go version 1.20 only (support win7)
 cd root directory of the socure code and execute
 ```
 make
@@ -136,14 +136,15 @@ Short-Term:
 6. Provide WebUI on client side.
 7. ~~Support private server, open source server program.~~(100%)
 8. Optimize our share scheduling model for different network operators.
-9. Provide REST APIs and libary for secondary development.
+9. ~~Provide REST APIs and libary for secondary development.~~(100%)
 10. ~~Support UDP at application layer, it is easy to implement but not urgent due to only a few applicaitons using UDP protocol.~~(100%)
-11. Support KCP protocol underlay, currently support Quic only. KCP focus on delay optimization,which has been widely used as game accelerator,it can sacrifice part of bandwidth to reduce timelag. 
+11. ~~Support KCP protocol underlay, currently support Quic only. KCP focus on delay optimization,which has been widely used as game accelerator,it can sacrifice part of bandwidth to reduce timelag. ~~(100%)
 12. ~~Support Android platform, let the phones to be mobile gateway.~~(100%)
-13. Support SMB Windows neighborhood.
-14. Direct connection on intranet, for testing.
+13. ~~Support SMB Windows neighborhood.~~(100%)
+14. ~~Direct connection on intranet, for testing.~~(100%)
 15. ~~Support UPNP.~~(100%)
-
+16. ~~Support Android~~(100%)
+17. Support IOS
 
 Long-Term:
 1. Use blockchain technology to decentralize, so that users who share equipment have benefits, thereby promoting more users to share, and achieving a positive closed loop.
@@ -161,3 +162,5 @@ Email: openp2p.cn@gmail.com tenderiron@139.com
 ## Disclaimer
 This project is open source for everyone to learn and use for free. It is forbidden to be used for illegal purposes. Any loss caused by improper use of this project or accident, this project and related personnel will not bear any responsibility. 
 
+## Thanks
+[![Powered by DartNode](https://dartnode.com/branding/DN-Open-Source-sm.png)](https://dartnode.com "Powered by DartNode - Free VPS for Open Source")

USAGE-ZH.md

@@ -43,7 +43,7 @@ nohup ./openp2p -d -node OFFICEPC1 -token TOKEN  &
 ```
 {
   "network": {
-    "Node": "hhd1207-222",
+    "Node": "YOUR-NODE-NAME",
     "Token": "TOKEN",
     "ShareBandwidth": 0,
     "ServerHost": "api.openp2p.cn",
@@ -87,7 +87,8 @@ systemctl stop firewalld.service
 systemctl start firewalld.service
 firewall-cmd --state
 ```
-
+## 停止
+TODO: windows linux macos
 ## 卸载
 ```
 ./openp2p uninstall

USAGE.md

@@ -46,7 +46,7 @@ Configuration example
 ```
 {
   "network": {
-    "Node": "hhd1207-222",
+    "Node": "YOUR-NODE-NAME",
     "Token": "TOKEN",
     "ShareBandwidth": 0,
     "ServerHost": "api.openp2p.cn",

app/.idea/runConfigurations.xml

@@ -1,10 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
-  <component name="RunConfigurationProducerService">
-    <option name="ignoredProducers">
-      <set>
-        <option value="com.android.tools.idea.compose.preview.runconfiguration.ComposePreviewRunConfigurationProducer" />
-      </set>
-    </option>
-  </component>
-</project>

app/README.md

@@ -1,7 +1,12 @@
 ## Build
+depends on openjdk 11, gradle 8.1.3, ndk 21
 ```
+
+# latest version not support go1.20
+go install golang.org/x/mobile/cmd/gomobile@7c4916698cc93475ebfea76748ee0faba2deb2a5
+gomobile init
+go get -v golang.org/x/mobile/bind@7c4916698cc93475ebfea76748ee0faba2deb2a5
 cd core
-go get -v golang.org/x/mobile/bind
 gomobile bind -target android -v
 if [[ $? -ne 0 ]]; then
     echo "build error"
@@ -11,6 +16,17 @@ echo "build ok"
 cp openp2p.aar openp2p-sources.jar ../app/app/libs
 echo "copy to APP libs"
 
+edit app/app/build.gradle 
+```
+signingConfigs {
+        release {
+            storeFile file('YOUR-JKS-PATH')
+            storePassword 'YOUR-PASSWORD'
+            keyAlias 'openp2p.keys'
+            keyPassword 'YOUR-PASSWORD'
+        }
+    }
+```
 cd ../app
 ./gradlew build
 

app/app/build.gradle

@@ -0,0 +1,63 @@
+plugins {
+    id 'com.android.application'
+    id 'kotlin-android'
+}
+
+android {
+    signingConfigs {
+        release {
+            storeFile file('C:\\work\\src\\openp2p-client\\app\\openp2p.jks')
+            storePassword 'YOUR-PASSWORD'
+            keyAlias 'openp2p.keys'
+            keyPassword 'YOUR-PASSWORD'
+        }
+    }
+    compileSdkVersion 31
+    buildToolsVersion "30.0.3"
+
+    defaultConfig {
+        applicationId "cn.openp2p"
+        minSdkVersion 16
+        targetSdkVersion 31
+        versionCode 1
+        versionName "2718281828"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+        namespace "cn.openp2p"
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled true
+            shrinkResources true
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+            signingConfig signingConfigs.release
+        }
+    }
+    compileOptions {
+        sourceCompatibility JavaVersion.VERSION_1_8
+        targetCompatibility JavaVersion.VERSION_1_8
+    }
+    kotlinOptions {
+        jvmTarget = '1.8'
+    }
+    buildFeatures {
+        viewBinding true
+    }
+}
+
+dependencies {
+    implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])
+    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
+    implementation 'androidx.core:core-ktx:1.3.1'
+    implementation 'androidx.appcompat:appcompat:1.2.0'
+    implementation 'com.google.android.material:material:1.2.1'
+    implementation 'androidx.annotation:annotation:1.1.0'
+    implementation 'androidx.constraintlayout:constraintlayout:2.0.1'
+    implementation 'androidx.lifecycle:lifecycle-livedata-ktx:2.2.0'
+    implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.2.0'
+    testImplementation 'junit:junit:4.+'
+    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
+    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'
+    implementation files('libs\\openp2p-sources.jar')
+}

app/app/src/main/AndroidManifest.xml

@@ -1,12 +1,11 @@
 <?xml version="1.0" encoding="utf-8"?>
-<manifest xmlns:android="http://schemas.android.com/apk/res/android"
-    package="cn.openp2p">
+<manifest xmlns:android="http://schemas.android.com/apk/res/android">
 
     <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
     <uses-permission android:name="android.permission.INTERNET" />
     <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
     <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
-
+    <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />
 
     <application
         android:allowBackup="true"
@@ -27,13 +26,21 @@
 
         <activity
             android:name=".ui.login.LoginActivity"
-            android:label="@string/app_name">
+            android:label="@string/app_name"
+            android:exported="true">
             <intent-filter>
                 <action android:name="android.intent.action.MAIN" />
 
                 <category android:name="android.intent.category.LAUNCHER" />
             </intent-filter>
         </activity>
+        <receiver android:name="BootReceiver"
+            android:exported="true"
+            android:enabled="true">
+            <intent-filter>
+                <action android:name="android.intent.action.BOOT_COMPLETED"/>
+            </intent-filter>
+        </receiver>
     </application>
 
 </manifest>

app/app/src/main/java/cn/openp2p/BootReceiver.kt

@@ -0,0 +1,27 @@
+package cn.openp2p
+
+import android.content.BroadcastReceiver
+import android.content.Context
+import android.content.Intent
+import android.net.VpnService
+import android.os.Build
+import android.os.Bundle
+import android.util.Log
+import cn.openp2p.ui.login.LoginActivity
+class BootReceiver : BroadcastReceiver() {
+    override fun onReceive(context: Context, intent: Intent) {
+//        Logger.log("pp onReceive "+intent.action.toString())
+        Log.i("onReceive","start "+intent.action.toString())
+        // if (Intent.ACTION_BOOT_COMPLETED == intent.action) {
+        //     Log.i("onReceive","match "+intent.action.toString())
+        //     VpnService.prepare(context)
+        //     val intent = Intent(context, OpenP2PService::class.java)
+        //     if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
+        //         context.startForegroundService(intent)
+        //     } else {
+        //         context.startService(intent)
+        //     }
+        // }
+        Log.i("onReceive","end "+intent.action.toString())
+    }
+}

app/app/src/main/java/cn/openp2p/OpenP2PService.kt

@@ -4,9 +4,12 @@ import android.app.*
 import android.content.Context
 import android.content.Intent
 import android.graphics.Color
+import java.io.IOException
+import android.net.VpnService
 import android.os.Binder
 import android.os.Build
 import android.os.IBinder
+import android.os.ParcelFileDescriptor
 import android.util.Log
 import androidx.annotation.RequiresApi
 import androidx.core.app.NotificationCompat
@@ -14,9 +17,22 @@ import cn.openp2p.ui.login.LoginActivity
 import kotlinx.coroutines.GlobalScope
 import kotlinx.coroutines.launch
 import openp2p.Openp2p
+import java.io.FileInputStream
+import java.io.FileOutputStream
+import java.nio.ByteBuffer
+import kotlinx.coroutines.*
+import org.json.JSONObject
+
+data class Node(val name: String, val ip: String, val resource: String? = null)
 
 
-class OpenP2PService : Service() {
+data class Network(
+    val id: Long,
+    val name: String,
+    val gateway: String,
+    val Nodes: List<Node>
+)
+class OpenP2PService : VpnService() {
     companion object {
         private val LOG_TAG = OpenP2PService::class.simpleName
     }
@@ -29,10 +45,12 @@ class OpenP2PService : Service() {
     private lateinit var network: openp2p.P2PNetwork
     private lateinit var mToken: String
     private var running:Boolean =true
+    private var sdwanRunning:Boolean =false
+    private var vpnInterface: ParcelFileDescriptor? = null
+    private  var sdwanJob: Job? = null
     override fun onCreate() {
         Log.i(LOG_TAG, "onCreate - Thread ID = " + Thread.currentThread().id)
-        var channelId: String? = null
-        channelId = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
+        var channelId = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
             createNotificationChannel("kim.hsl", "ForegroundService")
         } else {
             ""
@@ -41,7 +59,7 @@ class OpenP2PService : Service() {
 
         val pendingIntent = PendingIntent.getActivity(
             this, 0,
-            notificationIntent, 0
+            notificationIntent, PendingIntent.FLAG_IMMUTABLE
         )
 
         val notification = channelId?.let {
@@ -54,7 +72,7 @@ class OpenP2PService : Service() {
 
         startForeground(1337, notification)
         super.onCreate()
-
+        refreshSDWAN()
     }
 
     override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
@@ -62,23 +80,173 @@ class OpenP2PService : Service() {
             LOG_TAG,
             "onStartCommand - startId = " + startId + ", Thread ID = " + Thread.currentThread().id
         )
+        startOpenP2P(null)
         return super.onStartCommand(intent, flags, startId)
     }
 
     override fun onBind(p0: Intent?): IBinder? {
-
         val token = p0?.getStringExtra("token")
-        Log.i(LOG_TAG, "onBind - Thread ID = " + Thread.currentThread().id + token)
+        Log.i(LOG_TAG, "onBind token=$token")
+        startOpenP2P(token)
+        return binder
+    }
+
+    private fun startOpenP2P(token : String?): Boolean {
+        if (sdwanRunning) {
+            return true
+        }
+
+        Log.i(LOG_TAG, "startOpenP2P - Thread ID = " + Thread.currentThread().id + token)
+        val oldToken = Openp2p.getToken(getExternalFilesDir(null).toString())
+        Log.i(LOG_TAG, "startOpenP2P oldtoken=$oldToken newtoken=$token")
+        if (oldToken=="0" && token==null){
+            return false
+        }
+        sdwanRunning = true
+        //        runSDWAN()
         GlobalScope.launch {
-            network = Openp2p.runAsModule(getExternalFilesDir(null).toString(), token, 0, 1)
+            network = Openp2p.runAsModule(
+                getExternalFilesDir(null).toString(),
+                token,
+                0,
+                1
+            ) // /storage/emulated/0/Android/data/cn.openp2p/files/
             val isConnect = network.connect(30000)  // ms
-            Log.i(OpenP2PService.LOG_TAG, "login result: " + isConnect.toString());
+            Log.i(LOG_TAG, "login result: " + isConnect.toString());
             do {
                 Thread.sleep(1000)
-            }while(network.connect(30000)&&running)
+            } while (network.connect(30000) && running)
             stopSelf()
         }
-        return binder
+        return false
+    }
+
+    private fun refreshSDWAN() {
+        GlobalScope.launch {
+            Log.i(OpenP2PService.LOG_TAG, "refreshSDWAN start");
+            while (true) {
+                Log.i(OpenP2PService.LOG_TAG, "waiting new sdwan config");
+                val buf = ByteArray(4096)
+                val buffLen = Openp2p.getAndroidSDWANConfig(buf)
+                Log.i(OpenP2PService.LOG_TAG, "closing running sdwan instance");
+                sdwanRunning = false
+                vpnInterface?.close()
+                vpnInterface = null
+                Thread.sleep(10000)
+                runSDWAN(buf.copyOfRange(0,buffLen.toInt() ))
+            }
+            Log.i(OpenP2PService.LOG_TAG, "refreshSDWAN end");
+        }
+    }
+    private suspend fun readTunLoop() {
+        val inputStream = FileInputStream(vpnInterface?.fileDescriptor).channel
+        if (inputStream==null){
+            Log.i(OpenP2PService.LOG_TAG, "open FileInputStream error: ");
+            return
+        }
+        Log.d(LOG_TAG, "read tun loop start")
+        val buffer = ByteBuffer.allocate(4096)
+        val byteArrayRead = ByteArray(4096)
+        while (sdwanRunning) {
+            buffer.clear()
+            val readBytes = inputStream.read(buffer)
+            if (readBytes <= 0) {
+//                Log.i(OpenP2PService.LOG_TAG, "inputStream.read error: ")
+                delay(1)
+                continue
+            }
+            buffer.flip()
+            buffer.get(byteArrayRead,0,readBytes)
+            Log.i(OpenP2PService.LOG_TAG, String.format("Openp2p.androidRead: %d", readBytes))
+            Openp2p.androidRead(byteArrayRead, readBytes.toLong())
+
+        }
+        Log.d(LOG_TAG, "read tun loop end")
+    }
+
+    private fun runSDWAN(buf:ByteArray) {
+        sdwanRunning=true
+        sdwanJob=GlobalScope.launch(context = Dispatchers.IO) {
+            Log.i(OpenP2PService.LOG_TAG, "runSDWAN start:${buf.decodeToString()}");
+            try{
+                var builder = Builder()
+                val jsonObject = JSONObject(buf.decodeToString())
+                val id = jsonObject.getLong("id")
+                val name = jsonObject.getString("name")
+                val gateway = jsonObject.getString("gateway")
+                val nodesArray = jsonObject.getJSONArray("Nodes")
+
+                val nodesList = mutableListOf<JSONObject>()
+                for (i in 0 until nodesArray.length()) {
+                    nodesList.add(nodesArray.getJSONObject(i))
+                }
+
+                val myNodeName = Openp2p.getAndroidNodeName()
+                Log.i(OpenP2PService.LOG_TAG, "getAndroidNodeName:${myNodeName}");
+                val nodeList = nodesList.map {
+                    val nodeName = it.getString("name")
+                    val nodeIp = it.getString("ip")
+                    if (nodeName==myNodeName){
+                        builder.addAddress(nodeIp, 24)
+                    }
+                    val nodeResource = if (it.has("resource")) it.getString("resource") else null
+                    val parts = nodeResource?.split("/")
+                    if (parts?.size == 2) {
+                        val ipAddress = parts[0]
+                        val subnetMask = parts[1]
+                        builder.addRoute(ipAddress, subnetMask.toInt())
+                        Log.i(OpenP2PService.LOG_TAG, "sdwan addRoute:${ipAddress},${subnetMask.toInt()}");
+                    }
+                    Node(nodeName, nodeIp, nodeResource)
+                }
+
+                val network = Network(id, name, gateway, nodeList)
+                println(network)
+                Log.i(OpenP2PService.LOG_TAG, "onBind");
+                builder.addDnsServer("223.5.5.5")
+                builder.addDnsServer("2400:3200::1") // alicloud dns v6 & v4
+                builder.addRoute("10.2.3.0", 24)
+//            builder.addRoute("0.0.0.0", 0);
+                builder.setSession(LOG_TAG!!)
+                builder.setMtu(1420)
+                vpnInterface = builder.establish()
+                if (vpnInterface==null){
+                    Log.e(OpenP2PService.LOG_TAG, "start vpnservice error: ");
+                }
+               val outputStream = FileOutputStream(vpnInterface?.fileDescriptor).channel
+               if (outputStream==null){
+                   Log.e(OpenP2PService.LOG_TAG, "open FileOutputStream error: ");
+                   return@launch
+               }
+
+               val byteArrayWrite = ByteArray(4096)
+                launch {
+                        readTunLoop()
+                    }
+
+                Log.d(LOG_TAG, "write tun loop start")
+               while (sdwanRunning) {
+                   val len = Openp2p.androidWrite(byteArrayWrite)
+                   Log.i(OpenP2PService.LOG_TAG, String.format("Openp2p.androidWrite: %d",len));
+                   val writeBytes = outputStream?.write(ByteBuffer.wrap(byteArrayWrite))
+                   if (writeBytes != null && writeBytes <= 0) {
+                       Log.i(OpenP2PService.LOG_TAG, "outputStream?.write error: ");
+                       continue
+                   }
+               }
+               outputStream.close()
+// 关闭 VPN 接口
+                vpnInterface?.close()
+// 置空变量以释放资源
+                vpnInterface = null
+                Log.d(LOG_TAG, "write tun loop end")
+            }catch (e: Exception) {
+                // 捕获异常并记录
+                Log.e("VPN Connection", "发生异常: ${e.message}")
+            }
+            Log.i(OpenP2PService.LOG_TAG, "runSDWAN end");
+        }
+
     }
 
     override fun onDestroy() {
@@ -93,12 +261,13 @@ class OpenP2PService : Service() {
     }
     fun isConnected(): Boolean {
         if (!::network.isInitialized) return false
-        return network?.connect(1000)
+        return network.connect(1000)
     }
 
     fun stop() {
         running=false
         stopSelf()
+        Openp2p.stop()
     }
     @RequiresApi(Build.VERSION_CODES.O)
     private fun createNotificationChannel(channelId: String, channelName: String): String? {

app/app/src/main/java/cn/openp2p/log.kt

@@ -0,0 +1,45 @@
+package cn.openp2p
+import android.content.*
+import java.io.File
+import java.io.FileWriter
+import java.text.SimpleDateFormat
+import java.util.*
+import android.app.*
+import android.content.Context
+import android.content.Intent
+import android.graphics.Color
+import java.io.IOException
+import android.net.VpnService
+import android.os.Binder
+import android.os.Build
+import android.os.IBinder
+import android.os.ParcelFileDescriptor
+import android.util.Log
+import androidx.annotation.RequiresApi
+import androidx.core.app.NotificationCompat
+import cn.openp2p.ui.login.LoginActivity
+import kotlinx.coroutines.GlobalScope
+import kotlinx.coroutines.launch
+import openp2p.Openp2p
+import java.io.FileInputStream
+import java.io.FileOutputStream
+import java.nio.ByteBuffer
+import kotlinx.coroutines.*
+import org.json.JSONObject
+
+object Logger {
+    private val logFile: File = File("app.log")
+
+    fun log(message: String) {
+        val timestamp = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()).format(Date())
+        val logMessage = "$timestamp: $message\n"
+
+        try {
+            val fileWriter = FileWriter(logFile, true)
+            fileWriter.append(logMessage)
+            fileWriter.close()
+        } catch (e: Exception) {
+            e.printStackTrace()
+        }
+    }
+}

app/app/src/main/java/cn/openp2p/ui/login/LoginActivity.kt

@@ -5,6 +5,7 @@ import android.app.Activity
 import android.app.ActivityManager
 import android.app.Notification
 import android.app.PendingIntent
+import android.content.BroadcastReceiver
 import android.content.ComponentName
 import android.content.Context
 import android.content.Intent
@@ -25,6 +26,7 @@ import androidx.annotation.StringRes
 import androidx.appcompat.app.AppCompatActivity
 import androidx.lifecycle.Observer
 import androidx.lifecycle.ViewModelProvider
+import cn.openp2p.Logger
 import cn.openp2p.OpenP2PService
 import cn.openp2p.R
 import cn.openp2p.databinding.ActivityLoginBinding
@@ -51,6 +53,14 @@ class LoginActivity : AppCompatActivity() {
     private lateinit var loginViewModel: LoginViewModel
     private lateinit var binding: ActivityLoginBinding
     private lateinit var mService: OpenP2PService
+    @RequiresApi(Build.VERSION_CODES.O)
+    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
+        super.onActivityResult(requestCode, resultCode, data)
+        if (requestCode == 0 && resultCode == Activity.RESULT_OK) {
+            startService(Intent(this, OpenP2PService::class.java))
+        }
+    }
+
     @RequiresApi(Build.VERSION_CODES.O)
     override fun onCreate(savedInstanceState: Bundle?) {
         super.onCreate(savedInstanceState)
@@ -78,22 +88,18 @@ class LoginActivity : AppCompatActivity() {
                 token.error = getString(loginState.passwordError)
             }
         })
-        val intent1 = VpnService.prepare(this) ?: return
-        loginViewModel.loginResult.observe(this@LoginActivity, Observer {
-            val loginResult = it ?: return@Observer
-
-            loading.visibility = View.GONE
-            if (loginResult.error != null) {
-                showLoginFailed(loginResult.error)
+        openp2pLog.setText(R.string.phone_setting)
+        val intent = VpnService.prepare(this)
+         if (intent != null)
+         {
+             Log.i("openp2p", "VpnService.prepare need permission");
+             startActivityForResult(intent, 0)
          }
-            if (loginResult.success != null) {
-                updateUiWithUser(loginResult.success)
+         else {
+             Log.i("openp2p", "VpnService.prepare ready");
+             onActivityResult(0, Activity.RESULT_OK, null)
          }
-            setResult(Activity.RESULT_OK)
 
-            //Complete and destroy login activity once successful
-            finish()
-        })
 
         profile.setOnClickListener {
             val url = "https://console.openp2p.cn/profile"
@@ -110,17 +116,23 @@ class LoginActivity : AppCompatActivity() {
             }
 
             openp2pLog.setText(R.string.phone_setting)
-            token.setText(Openp2p.getToken(getExternalFilesDir(null).toString()))
+
             login.setOnClickListener {
                 if (login.text.toString()=="退出"){
-//                    val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
-//                    stopService(intent)
+                //    val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
+                //    stopService(intent)
                     Log.i(LOG_TAG, "quit")
                     mService.stop()
-                    unbindService(connection)
+
                     val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
                     stopService(intent)
+                    // 解绑服务
+                    unbindService(connection)
+
+                    // 结束当前 Activity
+                    finish() // 或者使用 finishAffinity() 来结束整个应用程序
                     exitAPP()
+                    // finishAffinity()
 
                 }
                 login.setText("退出")
@@ -139,13 +151,20 @@ class LoginActivity : AppCompatActivity() {
                             if (isConnect) {
                                 onlineState.setText("在线")
                             } else {
-                                onlineState.setText("离线")
+                                onlineState.setText("正在登录")
                             }
                         }
                     } while (true)
                 }
 
             }
+            val tokenText = Openp2p.getToken(getExternalFilesDir(null).toString())
+            token.setText(tokenText.toString())
+            // Check token length and automatically click login if length > 10
+            if (tokenText.length > 10) {
+//                Logger.log("performClick ")
+                login.performClick()
+            }
         }
     }
     @RequiresApi(Build.VERSION_CODES.LOLLIPOP)

app/app/src/main/res/layout/activity_login.xml

@@ -13,28 +13,27 @@
 
     <EditText
         android:id="@+id/token"
-        android:layout_width="225dp"
-        android:layout_height="46dp"
+        android:layout_width="250dp"
+        android:layout_height="45dp"
         android:hint="Token"
         android:imeActionLabel="@string/action_sign_in_short"
         android:imeOptions="actionDone"
         android:selectAllOnFocus="true"
         app:layout_constraintStart_toStartOf="parent"
-        app:layout_constraintTop_toTopOf="parent"
-         />
+        app:layout_constraintTop_toTopOf="parent" />
 
     <Button
         android:id="@+id/login"
-        android:layout_width="wrap_content"
-        android:layout_height="wrap_content"
+        android:layout_width="85dp"
+        android:layout_height="45dp"
         android:layout_gravity="start"
         android:layout_marginStart="24dp"
         android:layout_marginLeft="24dp"
         android:enabled="false"
         android:text="@string/action_sign_in"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toEndOf="@+id/token"
-        app:layout_constraintTop_toTopOf="parent"
-        tools:layout_editor_absoluteY="-2dp" />
+        app:layout_constraintTop_toTopOf="parent" />
 
 
 
@@ -58,28 +57,36 @@
         android:id="@+id/openp2pLog"
         android:layout_width="359dp"
         android:layout_height="548dp"
+        android:layout_marginTop="24dp"
         android:ems="10"
-        android:inputType="none"
-        android:textIsSelectable="true"
         android:focusable="false"
+        android:inputType="none"
         android:text="Name"
+        android:textIsSelectable="true"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toStartOf="parent"
         app:layout_constraintTop_toBottomOf="@+id/onlineState" />
+
     <Button
         android:id="@+id/profile"
-        android:layout_width="wrap_content"
-        android:layout_height="wrap_content"
+        android:layout_width="250dp"
+        android:layout_height="45dp"
+        android:layout_marginTop="8dp"
         android:text="打开控制台查看Token"
         app:layout_constraintStart_toStartOf="parent"
         app:layout_constraintTop_toBottomOf="@+id/token" />
 
     <EditText
         android:id="@+id/onlineState"
-        android:layout_width="113dp"
+        android:layout_width="85dp"
         android:layout_height="45dp"
+        android:layout_marginStart="24dp"
+        android:layout_marginLeft="24dp"
+        android:layout_marginTop="8dp"
         android:ems="10"
         android:inputType="textPersonName"
         android:text="未登录"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toEndOf="@+id/profile"
         app:layout_constraintTop_toBottomOf="@+id/login" />
 

app/app/src/main/res/values/strings.xml

@@ -10,6 +10,13 @@
     <string name="invalid_password">Token可以在 https://console.openp2p.cn/profile 获得</string>
     <string name="login_failed">"Login failed"</string>
     <string name="phone_setting">"安卓系统默认设置的”杀后台进程“会导致 OpenP2P 在后台运行一会后，被系统杀死进程，导致您的体验受到影响。您可以通过以下方式修改几个设置，解决此问题：
+华为鸿蒙：
+    1. 允许应用后台运行：进入设置 → 搜索进入 应用启动管理 → 关闭 OpenP2P 的 自动管理 开关 → 在弹框中勾选 允许后台活动 
+    2. 避免应用被电池优化程序清理：进入设置 → 搜索进入电池优化 → 不允许 →选择所有应用 → 找到无法后台运行的应用 → 设置为不允许
+    3. 关闭省电模式：进入设置 → 电池 → 关闭 省电模式 开关
+    4. 保持设备网络连接：进入设置  → 电池 → 更多电池设置 → 开启 休眠时始终保持网络连接 开关。
+    5. 给后台运行的应用加锁：打开应用后 → 进入多任务界面 → 下拉选中的卡片进行加锁 → 然后点击清理图标清理其他不经常使用的应用
+    6. 设置开发人员选项中相关开关：进入设置 → 搜索进入 开发人员选项 → 找到 不保留活动 开关后关闭 → 并在 后台进程限制 选择 标准限制
     
 华为手机：
     进入”设置“，搜索并进入“电池优化“界面，选中 OpenP2P 程序，不允许系统对其进行电池优化；

app/build.gradle

@@ -0,0 +1,30 @@
+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+
+buildscript {
+
+    ext.kotlin_version = "1.8.20"
+    repositories {
+        google()
+        mavenCentral()
+    }
+    dependencies {
+        classpath "com.android.tools.build:gradle:8.1.3"
+        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
+
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+
+allprojects {
+    repositories {
+        google()
+        mavenCentral()
+        jcenter() // Warning: this repository is going to shut down soon
+    }
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

app/gradle/wrapper/gradle-wrapper.properties

@@ -1,6 +1,6 @@
-#Sat Oct 22 21:46:24 CST 2022
+#Tue Dec 05 16:04:08 CST 2023
 distributionBase=GRADLE_USER_HOME
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.7.1-bin.zip
 distributionPath=wrapper/dists
-zipStorePath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-bin.zip
 zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists

cmd/openp2p.go

@@ -1,9 +1,9 @@
 package main
 
 import (
-	core "openp2p/core"
+	op "openp2p/core"
 )
 
 func main() {
-	core.Run()
+	op.Run()
 }

core/bandwidthLimit.go

@@ -1,45 +0,0 @@
-package openp2p
-
-import (
-	"sync"
-	"time"
-)
-
-// BandwidthLimiter ...
-type BandwidthLimiter struct {
-	ts           time.Time
-	bw           int // mbps
-	freeBytes    int // bytes
-	maxFreeBytes int // bytes
-	mtx          sync.Mutex
-}
-
-// mbps
-func newBandwidthLimiter(bw int) *BandwidthLimiter {
-	return &BandwidthLimiter{
-		bw:           bw,
-		ts:           time.Now(),
-		maxFreeBytes: bw * 1024 * 1024 / 8,
-		freeBytes:    bw * 1024 * 1024 / 8,
-	}
-}
-
-// Add ...
-func (bl *BandwidthLimiter) Add(bytes int) {
-	if bl.bw <= 0 {
-		return
-	}
-	bl.mtx.Lock()
-	defer bl.mtx.Unlock()
-	// calc free flow 1000*1000/1024/1024=0.954; 1024*1024/1000/1000=1.048
-	bl.freeBytes += int(time.Since(bl.ts) * time.Duration(bl.bw) / 8 / 954)
-	if bl.freeBytes > bl.maxFreeBytes {
-		bl.freeBytes = bl.maxFreeBytes
-	}
-	bl.freeBytes -= bytes
-	bl.ts = time.Now()
-	if bl.freeBytes < 0 {
-		// sleep for the overflow
-		time.Sleep(time.Millisecond * time.Duration(-bl.freeBytes/(bl.bw*1048/8)))
-	}
-}

core/common.go

@@ -5,8 +5,10 @@ import (
 	"crypto/aes"
 	"crypto/cipher"
 	"crypto/tls"
+	"encoding/binary"
 	"encoding/json"
 	"fmt"
+	"math/big"
 	"math/rand"
 	"net"
 	"net/http"
@@ -197,8 +199,12 @@ func parseMajorVer(ver string) int {
 	return 0
 }
 
-func IsIPv6(address string) bool {
-	return strings.Count(address, ":") >= 2
+func IsIPv6(ipStr string) bool {
+	ip := net.ParseIP(ipStr)
+	if ip == nil {
+		return false
+	}
+	return ip.To16() != nil && ip.To4() == nil
 }
 
 var letters = []byte("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-")
@@ -210,3 +216,67 @@ func randStr(n int) string {
 	}
 	return string(b)
 }
+
+func execCommand(commandPath string, wait bool, arg ...string) (err error) {
+	command := exec.Command(commandPath, arg...)
+	err = command.Start()
+	if err != nil {
+		return
+	}
+	if wait {
+		err = command.Wait()
+	}
+	return
+}
+
+func sanitizeFileName(fileName string) string {
+	validFileName := fileName
+	invalidChars := []string{"\\", "/", ":", "*", "?", "\"", "<", ">", "|"}
+	for _, char := range invalidChars {
+		validFileName = strings.ReplaceAll(validFileName, char, " ")
+	}
+	return validFileName
+}
+
+func prettyJson(s interface{}) string {
+	jsonData, err := json.MarshalIndent(s, "", "  ")
+	if err != nil {
+		fmt.Println("Error marshalling JSON:", err)
+		return ""
+	}
+	return string(jsonData)
+}
+
+func inetAtoN(ipstr string) (uint32, error) { // support both ipnet or single ip
+	i, _, err := net.ParseCIDR(ipstr)
+	if err != nil {
+		i = net.ParseIP(ipstr)
+		if i == nil {
+			return 0, err
+		}
+	}
+	ret := big.NewInt(0)
+	ret.SetBytes(i.To4())
+	return uint32(ret.Int64()), nil
+}
+
+func calculateChecksum(data []byte) uint16 {
+	length := len(data)
+	sum := uint32(0)
+
+	// Calculate the sum of 16-bit words
+	for i := 0; i < length-1; i += 2 {
+		sum += uint32(binary.BigEndian.Uint16(data[i : i+2]))
+	}
+
+	// Add the last byte (if odd length)
+	if length%2 != 0 {
+		sum += uint32(data[length-1])
+	}
+
+	// Fold 32-bit sum to 16 bits
+	sum = (sum >> 16) + (sum & 0xffff)
+	sum += (sum >> 16)
+
+	return uint16(^sum)
+}

core/common_test.go

@@ -1,6 +1,7 @@
 package openp2p
 
 import (
+	"fmt"
 	"log"
 	"testing"
 )
@@ -94,3 +95,35 @@ func TestParseMajorVer(t *testing.T) {
 	assertParseMajorVer(t, "3.0.0", 3)
 
 }
+
+func TestIsIPv6(t *testing.T) {
+	tests := []struct {
+		ipStr string
+		want  bool
+	}{
+		{"2001:0db8:85a3:0000:0000:8a2e:0370:7334", true}, // 有效的 IPv6 地址
+		{"2001:db8::2:1", true},                           // 有效的 IPv6 地址
+		{"192.168.1.1", false},                            // 无效的 IPv6 地址，是 IPv4
+		{"2001:db8::G:1", false},                          // 无效的 IPv6 地址，包含非法字符
+		// 可以添加更多测试用例
+	}
+
+	for _, tt := range tests {
+		got := IsIPv6(tt.ipStr)
+		if got != tt.want {
+			t.Errorf("isValidIPv6(%s) = %v, want %v", tt.ipStr, got, tt.want)
+		}
+	}
+}
+
+func TestNodeID(t *testing.T) {
+	node1 := "n1-stable"
+	node2 := "tony-stable"
+	nodeID1 := NodeNameToID(node1)
+	nodeID2 := NodeNameToID(node2)
+	if nodeID1 < nodeID2 {
+		fmt.Printf("%s < %s\n", node1, node2)
+	} else {
+		fmt.Printf("%s >= %s\n", node1, node2)
+	}
+}

core/config.go

@@ -4,9 +4,9 @@ import (
 	"encoding/json"
 	"flag"
 	"fmt"
-	"io/ioutil"
 	"os"
 	"strconv"
+	"strings"
 	"sync"
 	"time"
 )
@@ -17,6 +17,8 @@ type AppConfig struct {
 	// required
 	AppName          string
 	Protocol         string
+	UnderlayProtocol string
+	PunchPriority    int // bitwise DisableTCP|DisableUDP|TCPFirst  0:tcp and udp both enable, udp first
 	Whitelist        string
 	SrcPort          int
 	PeerNode         string
@@ -24,11 +26,14 @@ type AppConfig struct {
 	DstHost          string
 	PeerUser         string
 	RelayNode        string
+	ForceRelay       int // default:0 disable;1 enable
 	Enabled          int // default:1
 	// runtime info
+	relayMode        string // private|public
 	peerVersion      string
 	peerToken        uint64
 	peerNatType      int
+	peerLanIP        string
 	hasIPv4          int
 	peerIPv6         string
 	hasUPNPorNATPMP  int
@@ -42,20 +47,103 @@ type AppConfig struct {
 	connectTime      time.Time
 	fromToken        uint64
 	linkMode         string
-	isUnderlayServer int // TODO: bool?
+	isUnderlayServer int
 }
 
-func (c *AppConfig) ID() string {
-	return fmt.Sprintf("%s%d", c.Protocol, c.SrcPort)
+const (
+	PunchPriorityTCPFirst   = 1
+	PunchPriorityUDPDisable = 1 << 1
+	PunchPriorityTCPDisable = 1 << 2
+)
+
+func (c *AppConfig) ID() uint64 {
+	if c.SrcPort == 0 { // memapp
+		return NodeNameToID(c.PeerNode)
+	}
+	if c.Protocol == "tcp" {
+		return uint64(c.SrcPort) * 10
+	}
+	return uint64(c.SrcPort)*10 + 1
+}
+
+func (c *AppConfig) LogPeerNode() string {
+	if c.relayMode == "public" { // memapp
+		return fmt.Sprintf("%d", NodeNameToID(c.PeerNode))
+	}
+	return c.PeerNode
 }
 
-// TODO: add loglevel, maxlogfilesize
 type Config struct {
 	Network NetworkConfig `json:"network"`
 	Apps    []*AppConfig  `json:"apps"`
+
 	LogLevel   int
+	MaxLogSize int
 	daemonMode bool
 	mtx        sync.Mutex
+	sdwanMtx   sync.Mutex
+	sdwan      SDWANInfo
+	delNodes   []*SDWANNode
+	addNodes   []*SDWANNode
+}
+
+func (c *Config) getSDWAN() SDWANInfo {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.sdwan
+}
+
+func (c *Config) getDelNodes() []*SDWANNode {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.delNodes
+}
+
+func (c *Config) getAddNodes() []*SDWANNode {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.addNodes
+}
+
+func (c *Config) resetSDWAN() {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	c.delNodes = []*SDWANNode{}
+	c.addNodes = []*SDWANNode{}
+	c.sdwan = SDWANInfo{}
+}
+func (c *Config) setSDWAN(s SDWANInfo) {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	// get old-new
+	c.delNodes = []*SDWANNode{}
+	for _, oldNode := range c.sdwan.Nodes {
+		isDeleted := true
+		for _, newNode := range s.Nodes {
+			if oldNode.Name == newNode.Name && oldNode.IP == newNode.IP && oldNode.Resource == newNode.Resource && c.sdwan.Mode == s.Mode && c.sdwan.CentralNode == s.CentralNode {
+				isDeleted = false
+				break
+			}
+		}
+		if isDeleted {
+			c.delNodes = append(c.delNodes, oldNode)
+		}
+	}
+	// get new-old
+	c.addNodes = []*SDWANNode{}
+	for _, newNode := range s.Nodes {
+		isNew := true
+		for _, oldNode := range c.sdwan.Nodes {
+			if oldNode.Name == newNode.Name && oldNode.IP == newNode.IP && oldNode.Resource == newNode.Resource && c.sdwan.Mode == s.Mode && c.sdwan.CentralNode == s.CentralNode {
+				isNew = false
+				break
+			}
+		}
+		if isNew {
+			c.addNodes = append(c.addNodes, newNode)
+		}
+	}
+	c.sdwan = s
 }
 
 func (c *Config) switchApp(app AppConfig, enabled int) {
@@ -71,28 +159,72 @@ func (c *Config) switchApp(app AppConfig, enabled int) {
 	}
 	c.save()
 }
+
 func (c *Config) retryApp(peerNode string) {
-	c.mtx.Lock()
-	defer c.mtx.Unlock()
-	for i := 0; i < len(c.Apps); i++ {
-		if c.Apps[i].PeerNode == peerNode {
-			c.Apps[i].retryNum = 0
-			c.Apps[i].nextRetryTime = time.Now()
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		if app.config.PeerNode == peerNode {
+			gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+			app.config.retryNum = 0
+			app.config.nextRetryTime = time.Now()
+			app.retryRelayNum = 0
+			app.nextRetryRelayTime = time.Now()
+			app.hbMtx.Lock()
+			app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+			app.hbMtx.Unlock()
 		}
+		if app.config.RelayNode == peerNode {
+			gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+			app.retryRelayNum = 0
+			app.nextRetryRelayTime = time.Now()
+			app.hbMtx.Lock()
+			app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+			app.hbMtx.Unlock()
 		}
+		return true
+	})
+}
+
+func (c *Config) retryAllApp() {
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+		app.config.retryNum = 0
+		app.config.nextRetryTime = time.Now()
+		app.retryRelayNum = 0
+		app.nextRetryRelayTime = time.Now()
+		app.hbMtx.Lock()
+		defer app.hbMtx.Unlock()
+		app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+		return true
+	})
+}
+
+func (c *Config) retryAllMemApp() {
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		if app.config.SrcPort != 0 {
+			return true
+		}
+		gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+		app.config.retryNum = 0
+		app.config.nextRetryTime = time.Now()
+		app.retryRelayNum = 0
+		app.nextRetryRelayTime = time.Now()
+		app.hbMtx.Lock()
+		defer app.hbMtx.Unlock()
+		app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+		return true
+	})
 }
 
 func (c *Config) add(app AppConfig, override bool) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
 	defer c.save()
-	if app.SrcPort == 0 || app.DstPort == 0 {
-		gLog.Println(LvERROR, "invalid app ", app)
-		return
-	}
 	if override {
 		for i := 0; i < len(c.Apps); i++ {
-			if c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
+			if c.Apps[i].PeerNode == app.PeerNode && c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
 				c.Apps[i] = &app // override it
 				return
 			}
@@ -102,33 +234,51 @@ func (c *Config) add(app AppConfig, override bool) {
 }
 
 func (c *Config) delete(app AppConfig) {
-	if app.SrcPort == 0 || app.DstPort == 0 {
-		return
-	}
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
 	defer c.save()
 	for i := 0; i < len(c.Apps); i++ {
-		if c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
+		if (app.SrcPort != 0 && c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort) || // normal app
+			(app.SrcPort == 0 && c.Apps[i].SrcPort == 0 && c.Apps[i].PeerNode == app.PeerNode) { // memapp
+			if i == len(c.Apps)-1 {
+				c.Apps = c.Apps[:i]
+			} else {
 				c.Apps = append(c.Apps[:i], c.Apps[i+1:]...)
+			}
 			return
 		}
 	}
-
 }
 
 func (c *Config) save() {
 	// c.mtx.Lock()
 	// defer c.mtx.Unlock()  // internal call
+	if c.Network.Token == 0 {
+		return
+	}
 	data, _ := json.MarshalIndent(c, "", "  ")
-	err := ioutil.WriteFile("config.json", data, 0644)
+	err := os.WriteFile("config.json", data, 0644)
 	if err != nil {
 		gLog.Println(LvERROR, "save config.json error:", err)
 	}
 }
 
+func (c *Config) saveCache() {
+	// c.mtx.Lock()
+	// defer c.mtx.Unlock()  // internal call
+	if c.Network.Token == 0 {
+		return
+	}
+	data, _ := json.MarshalIndent(c, "", "  ")
+	err := os.WriteFile("config.json0", data, 0644)
+	if err != nil {
+		gLog.Println(LvERROR, "save config.json0 error:", err)
+	}
+}
+
 func init() {
 	gConf.LogLevel = int(LvINFO)
+	gConf.MaxLogSize = 1024 * 1024
 	gConf.Network.ShareBandwidth = 10
 	gConf.Network.ServerHost = "api.openp2p.cn"
 	gConf.Network.ServerPort = WsPort
@@ -138,19 +288,41 @@ func init() {
 func (c *Config) load() error {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
-	data, err := ioutil.ReadFile("config.json")
+	data, err := os.ReadFile("config.json")
 	if err != nil {
-		// gLog.Println(LevelERROR, "read config.json error:", err)
-		return err
+		return c.loadCache()
 	}
 	err = json.Unmarshal(data, &c)
 	if err != nil {
 		gLog.Println(LvERROR, "parse config.json error:", err)
+		// try cache
+		return c.loadCache()
+	}
+	// load ok. cache it
+	var filteredApps []*AppConfig // filter memapp
+	for _, app := range c.Apps {
+		if app.SrcPort != 0 {
+			filteredApps = append(filteredApps, app)
+		}
+	}
+	c.Apps = filteredApps
+	c.saveCache()
+	return err
+}
+
+func (c *Config) loadCache() error {
+	data, err := os.ReadFile("config.json0")
+	if err != nil {
+		return err
+	}
+	err = json.Unmarshal(data, &c)
+	if err != nil {
+		gLog.Println(LvERROR, "parse config.json0 error:", err)
 	}
 	return err
 }
 
-// TODO: deal with multi-thread r/w
+// deal with multi-thread r/w
 func (c *Config) setToken(token uint64) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
@@ -170,6 +342,15 @@ func (c *Config) setNode(node string) {
 	defer c.mtx.Unlock()
 	defer c.save()
 	c.Network.Node = node
+	c.Network.nodeID = NodeNameToID(c.Network.Node)
+}
+func (c *Config) nodeID() uint64 {
+	c.mtx.Lock()
+	defer c.mtx.Unlock()
+	if c.Network.nodeID == 0 {
+		c.Network.nodeID = NodeNameToID(c.Network.Node)
+	}
+	return c.Network.nodeID
 }
 func (c *Config) setShareBandwidth(bw int) {
 	c.mtx.Lock()
@@ -192,6 +373,7 @@ type NetworkConfig struct {
 	// local info
 	Token           uint64
 	Node            string
+	nodeID          uint64
 	User            string
 	localIP         string
 	mac             string
@@ -210,7 +392,7 @@ type NetworkConfig struct {
 	TCPPort    int
 }
 
-func parseParams(subCommand string) {
+func parseParams(subCommand string, cmd string) {
 	fset := flag.NewFlagSet(subCommand, flag.ExitOnError)
 	serverHost := fset.String("serverhost", "api.openp2p.cn", "server host ")
 	serverPort := fset.Int("serverport", WsPort, "server port ")
@@ -224,6 +406,8 @@ func parseParams(subCommand string) {
 	srcPort := fset.Int("srcport", 0, "source port ")
 	tcpPort := fset.Int("tcpport", 0, "tcp port for upnp or publicip")
 	protocol := fset.String("protocol", "tcp", "tcp or udp")
+	underlayProtocol := fset.String("underlay_protocol", "quic", "quic or kcp")
+	punchPriority := fset.Int("punch_priority", 0, "bitwise DisableTCP|DisableUDP|UDPFirst  0:tcp and udp both enable, tcp first")
 	appName := fset.String("appname", "", "app name")
 	relayNode := fset.String("relaynode", "", "relaynode")
 	shareBandwidth := fset.Int("sharebandwidth", 10, "N mbps share bandwidth limit, private network no limit")
@@ -231,12 +415,20 @@ func parseParams(subCommand string) {
 	notVerbose := fset.Bool("nv", false, "not log console")
 	newconfig := fset.Bool("newconfig", false, "not load existing config.json")
 	logLevel := fset.Int("loglevel", 1, "0:debug 1:info 2:warn 3:error")
+	maxLogSize := fset.Int("maxlogsize", 1024*1024, "default 1MB")
+	if cmd == "" {
 		if subCommand == "" { // no subcommand
 			fset.Parse(os.Args[1:])
 		} else {
 			fset.Parse(os.Args[2:])
 		}
+	} else {
+		gLog.Println(LvINFO, "cmd=", cmd)
+		args := strings.Split(cmd, " ")
+		fset.Parse(args)
+	}
 
+	gLog.setMaxSize(int64(*maxLogSize))
 	config := AppConfig{Enabled: 1}
 	config.PeerNode = *peerNode
 	config.DstHost = *dstIP
@@ -244,12 +436,14 @@ func parseParams(subCommand string) {
 	config.DstPort = *dstPort
 	config.SrcPort = *srcPort
 	config.Protocol = *protocol
+	config.UnderlayProtocol = *underlayProtocol
+	config.PunchPriority = *punchPriority
 	config.AppName = *appName
 	config.RelayNode = *relayNode
 	if !*newconfig {
 		gConf.load() // load old config. otherwise will clear all apps
 	}
-	if config.SrcPort != 0 {
+	if config.SrcPort != 0 { // filter memapp
 		gConf.add(config, true)
 	}
 	// gConf.mtx.Lock() // when calling this func it's single-thread no lock
@@ -260,7 +454,7 @@ func parseParams(subCommand string) {
 			gConf.Network.ShareBandwidth = *shareBandwidth
 		}
 		if f.Name == "node" {
-			gConf.Network.Node = *node
+			gConf.setNode(*node)
 		}
 		if f.Name == "serverhost" {
 			gConf.Network.ServerHost = *serverHost
@@ -268,6 +462,9 @@ func parseParams(subCommand string) {
 		if f.Name == "loglevel" {
 			gConf.LogLevel = *logLevel
 		}
+		if f.Name == "maxlogsize" {
+			gConf.MaxLogSize = *maxLogSize
+		}
 		if f.Name == "tcpport" {
 			gConf.Network.TCPPort = *tcpPort
 		}
@@ -280,19 +477,19 @@ func parseParams(subCommand string) {
 		gConf.Network.ServerHost = *serverHost
 	}
 	if *node != "" {
-		gConf.Network.Node = *node
+		gConf.setNode(*node)
 	} else {
 		envNode := os.Getenv("OPENP2P_NODE")
 		if envNode != "" {
-			gConf.Network.Node = envNode
+			gConf.setNode(envNode)
 		}
 		if gConf.Network.Node == "" { // if node name not set. use os.Hostname
-			gConf.Network.Node = defaultNodeName()
+			gConf.setNode(defaultNodeName())
 		}
 	}
 	if gConf.Network.TCPPort == 0 {
 		if *tcpPort == 0 {
-			p := int(nodeNameToID(gConf.Network.Node)%15000 + 50000)
+			p := int(gConf.nodeID()%15000 + 50000)
 			tcpPort = &p
 		}
 		gConf.Network.TCPPort = *tcpPort

core/config_test.go

@@ -0,0 +1,178 @@
+package openp2p
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func TestSetSDWAN_ChangeNode(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	if len(conf.getAddNodes()) != 7 {
+		t.Errorf("getAddNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo3 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo3); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo3)
+	diff = conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.60" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	// add new node
+	sdwanInfo4 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo4); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo4)
+	diff = conf.getDelNodes()
+	if len(diff) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.60" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+
+func TestSetSDWAN_ChangeNodeIP(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.44","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 || diff[0].IP != "10.2.3.44" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+func TestSetSDWAN_ClearAll(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"Nodes":null}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 7 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 0 {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+func TestSetSDWAN_ChangeNodeResource(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.11.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 || diff[0].Resource != "10.11.0.0/16" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+
+func TestInetAtoN(t *testing.T) {
+	ipa, _ := inetAtoN("121.5.147.4")
+	t.Log(ipa)
+	ipa, _ = inetAtoN("121.5.147.4/32")
+	t.Log(ipa)
+}

core/daemon.go

@@ -60,6 +60,7 @@ func (d *daemon) run() {
 			break
 		}
 	}
+
 	args = append(args, "-nv")
 	for {
 		// start worker
@@ -72,6 +73,7 @@ func (d *daemon) run() {
 		}
 		gLog.Println(LvINFO, "start worker process, args:", args)
 		execSpec := &os.ProcAttr{Env: append(os.Environ(), "GOTRACEBACK=crash"), Files: []*os.File{os.Stdin, os.Stdout, f}}
+		lastRebootTime := time.Now()
 		p, err := os.StartProcess(binPath, args, execSpec)
 		if err != nil {
 			gLog.Printf(LvERROR, "start worker error:%s", err)
@@ -88,9 +90,12 @@ func (d *daemon) run() {
 		if !d.running {
 			return
 		}
+		if time.Since(lastRebootTime) < time.Second*10 {
 			gLog.Printf(LvERROR, "worker stop, restart it after 10s")
 			time.Sleep(time.Second * 10)
 		}
+
+	}
 }
 
 func (d *daemon) Control(ctrlComm string, exeAbsPath string, args []string) error {

core/errorcode.go

@@ -13,10 +13,21 @@ var (
 	ErrorNewUser             = errors.New("new user")
 	ErrorLogin               = errors.New("user or password not correct")
 	ErrNodeTooShort          = errors.New("node name too short, it must >=8 charaters")
+	ErrReadDB                = errors.New("read db error")
+	ErrNoUpdate              = errors.New("there are currently no updates available")
 	ErrPeerOffline           = errors.New("peer offline")
 	ErrNetwork               = errors.New("network error")
 	ErrMsgFormat             = errors.New("message format wrong")
 	ErrVersionNotCompatible  = errors.New("version not compatible")
 	ErrOverlayConnDisconnect = errors.New("overlay connection is disconnected")
 	ErrConnectRelayNode      = errors.New("connect relay node error")
+	ErrConnectPublicV4       = errors.New("connect public ipv4 error")
+	ErrMsgChannelNotFound    = errors.New("message channel not found")
+	ErrRelayTunnelNotFound   = errors.New("relay tunnel not found")
+	ErrSymmetricLimit        = errors.New("symmetric limit")
+	ErrForceRelay            = errors.New("force relay")
+	ErrPeerConnectRelay      = errors.New("peer connect relayNode error")
+	ErrBuildTunnelBusy       = errors.New("build tunnel busy")
+	ErrMemAppTunnelNotFound  = errors.New("memapp tunnel not found")
+	ErrRemoteServiceUnable   = errors.New("remote service unable")
 )

core/handlepush.go

@@ -5,15 +5,17 @@ import (
 	"encoding/binary"
 	"encoding/json"
 	"fmt"
+	"net"
 	"os"
 	"path/filepath"
 	"reflect"
+	"runtime"
 	"time"
 
 	"github.com/openp2p-cn/totp"
 )
 
-func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
+func handlePush(subType uint16, msg []byte) error {
 	pushHead := PushHeader{}
 	err := binary.Read(bytes.NewReader(msg[openP2PHeaderSize:openP2PHeaderSize+PushHeaderSize]), binary.LittleEndian, &pushHead)
 	if err != nil {
@@ -21,8 +23,8 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 	}
 	gLog.Printf(LvDEBUG, "handle push msg type:%d, push header:%+v", subType, pushHead)
 	switch subType {
-	case MsgPushConnectReq: // TODO: handle a msg move to a new function
-		err = handleConnectReq(pn, subType, msg)
+	case MsgPushConnectReq:
+		err = handleConnectReq(msg)
 	case MsgPushRsp:
 		rsp := PushRsp{}
 		if err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp); err != nil {
@@ -43,16 +45,79 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		config := AppConfig{}
 		config.PeerNode = req.RelayName
 		config.peerToken = req.RelayToken
+		config.relayMode = req.RelayMode
 		go func(r AddRelayTunnelReq) {
-			t, errDt := pn.addDirectTunnel(config, 0)
+			t, errDt := GNetwork.addDirectTunnel(config, 0)
 			if errDt == nil {
 				// notify peer relay ready
 				msg := TunnelMsg{ID: t.id}
-				pn.push(r.From, MsgPushAddRelayTunnelRsp, msg)
+				GNetwork.push(r.From, MsgPushAddRelayTunnelRsp, msg)
+				appConfig := config
+				appConfig.PeerNode = req.From
 			} else {
-				pn.push(r.From, MsgPushAddRelayTunnelRsp, "error") // compatible with old version client, trigger unmarshal error
+				gLog.Printf(LvERROR, "addDirectTunnel error:%s", errDt)
+				GNetwork.push(r.From, MsgPushAddRelayTunnelRsp, "error") // compatible with old version client, trigger unmarshal error
 			}
 		}(req)
+	case MsgPushServerSideSaveMemApp:
+		req := ServerSideSaveMemApp{}
+		if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
+			gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
+			return err
+		}
+		gLog.Println(LvDEBUG, "handle MsgPushServerSideSaveMemApp:", prettyJson(req))
+		var existTunnel *P2PTunnel
+		i, ok := GNetwork.allTunnels.Load(req.TunnelID)
+		if !ok {
+			time.Sleep(time.Millisecond * 100)
+			i, ok = GNetwork.allTunnels.Load(req.TunnelID) // retry sometimes will receive MsgPushServerSideSaveMemApp but p2ptunnel not store yet.
+			if !ok {
+				gLog.Println(LvERROR, "handle MsgPushServerSideSaveMemApp error:", ErrMemAppTunnelNotFound)
+				return ErrMemAppTunnelNotFound
+			}
+		}
+		existTunnel = i.(*P2PTunnel)
+		peerID := NodeNameToID(req.From)
+		existApp, appok := GNetwork.apps.Load(peerID)
+		if appok {
+			app := existApp.(*p2pApp)
+			app.config.AppName = fmt.Sprintf("%d", peerID)
+			app.id = req.AppID
+			app.setRelayTunnelID(req.RelayTunnelID)
+			app.relayMode = req.RelayMode
+			app.hbTimeRelay = time.Now()
+			if req.RelayTunnelID == 0 {
+				app.setDirectTunnel(existTunnel)
+			} else {
+				app.setRelayTunnel(existTunnel)
+			}
+			gLog.Println(LvDEBUG, "find existing memapp, update it")
+		} else {
+			appConfig := existTunnel.config
+			appConfig.SrcPort = 0
+			appConfig.Protocol = ""
+			appConfig.AppName = fmt.Sprintf("%d", peerID)
+			appConfig.PeerNode = req.From
+			app := p2pApp{
+				id:          req.AppID,
+				config:      appConfig,
+				relayMode:   req.RelayMode,
+				running:     true,
+				hbTimeRelay: time.Now(),
+			}
+			if req.RelayTunnelID == 0 {
+				app.setDirectTunnel(existTunnel)
+			} else {
+				app.setRelayTunnel(existTunnel)
+				app.setRelayTunnelID(req.RelayTunnelID)
+			}
+			if req.RelayTunnelID != 0 {
+				app.relayNode = req.Node
+			}
+			GNetwork.apps.Store(NodeNameToID(req.From), &app)
+		}
+
+		return nil
 	case MsgPushAPPKey:
 		req := APPKeySync{}
 		if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
@@ -62,7 +127,7 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		SaveKey(req.AppID, req.AppKey)
 	case MsgPushUpdate:
 		gLog.Println(LvINFO, "MsgPushUpdate")
-		err := update(pn.config.ServerHost, pn.config.ServerPort)
+		err := update(gConf.Network.ServerHost, gConf.Network.ServerPort)
 		if err == nil {
 			os.Exit(0)
 		}
@@ -72,11 +137,17 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		os.Exit(0)
 		return err
 	case MsgPushReportApps:
-		err = handleReportApps(pn, subType, msg)
+		err = handleReportApps()
+	case MsgPushReportMemApps:
+		err = handleReportMemApps()
 	case MsgPushReportLog:
-		err = handleLog(pn, subType, msg)
+		err = handleLog(msg)
+	case MsgPushReportGoroutine:
+		err = handleReportGoroutine()
+	case MsgPushCheckRemoteService:
+		err = handleCheckRemoteService(msg)
 	case MsgPushEditApp:
-		err = handleEditApp(pn, subType, msg)
+		err = handleEditApp(msg)
 	case MsgPushEditNode:
 		gLog.Println(LvINFO, "MsgPushEditNode")
 		req := EditNode{}
@@ -86,7 +157,6 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		}
 		gConf.setNode(req.NewName)
 		gConf.setShareBandwidth(req.Bandwidth)
-		// TODO: hot reload
 		os.Exit(0)
 	case MsgPushSwitchApp:
 		gLog.Println(LvINFO, "MsgPushSwitchApp")
@@ -100,7 +170,7 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		gConf.switchApp(config, app.Enabled)
 		if app.Enabled == 0 {
 			// disable APP
-			pn.DeleteApp(config)
+			GNetwork.DeleteApp(config)
 		}
 	case MsgPushDstNodeOnline:
 		gLog.Println(LvINFO, "MsgPushDstNodeOnline")
@@ -112,15 +182,17 @@ func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
 		gLog.Println(LvINFO, "retry peerNode ", req.Node)
 		gConf.retryApp(req.Node)
 	default:
-		pn.msgMapMtx.Lock()
-		ch := pn.msgMap[pushHead.From]
-		pn.msgMapMtx.Unlock()
-		ch <- pushMsg{data: msg, ts: time.Now()}
+		i, ok := GNetwork.msgMap.Load(pushHead.From)
+		if !ok {
+			return ErrMsgChannelNotFound
+		}
+		ch := i.(chan msgCtx)
+		ch <- msgCtx{data: msg, ts: time.Now()}
 	}
 	return err
 }
 
-func handleEditApp(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
+func handleEditApp(msg []byte) (err error) {
 	gLog.Println(LvINFO, "MsgPushEditApp")
 	newApp := AppInfo{}
 	if err = json.Unmarshal(msg[openP2PHeaderSize:], &newApp); err != nil {
@@ -136,21 +208,24 @@ func handleEditApp(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 	oldConf.PeerNode = newApp.PeerNode
 	oldConf.DstHost = newApp.DstHost
 	oldConf.DstPort = newApp.DstPort
-
+	if newApp.Protocol0 != "" && newApp.SrcPort0 != 0 { // not edit
 		gConf.delete(oldConf)
+	}
+
 	// AddApp
 	newConf := oldConf
 	newConf.Protocol = newApp.Protocol
 	newConf.SrcPort = newApp.SrcPort
+	newConf.RelayNode = newApp.SpecRelayNode
+	newConf.PunchPriority = newApp.PunchPriority
 	gConf.add(newConf, false)
-	pn.DeleteApp(oldConf) // DeleteApp may cost some times, execute at the end
+	if newApp.Protocol0 != "" && newApp.SrcPort0 != 0 { // not edit
+		GNetwork.DeleteApp(oldConf) // DeleteApp may cost some times, execute at the end
+	}
 	return nil
-	// autoReconnect will auto AddApp
-	// pn.AddApp(config)
-	// TODO: report result
 }
 
-func handleConnectReq(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
+func handleConnectReq(msg []byte) (err error) {
 	req := PushConnectReq{}
 	if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
 		gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
@@ -158,21 +233,21 @@ func handleConnectReq(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 	}
 	gLog.Printf(LvDEBUG, "%s is connecting...", req.From)
 	gLog.Println(LvDEBUG, "push connect response to ", req.From)
-	if compareVersion(req.Version, LeastSupportVersion) == LESS {
+	if compareVersion(req.Version, LeastSupportVersion) < 0 {
 		gLog.Println(LvERROR, ErrVersionNotCompatible.Error(), ":", req.From)
 		rsp := PushConnectRsp{
 			Error:  10,
 			Detail: ErrVersionNotCompatible.Error(),
 			To:     req.From,
-			From:   pn.config.Node,
+			From:   gConf.Network.Node,
 		}
-		pn.push(req.From, MsgPushConnectRsp, rsp)
+		GNetwork.push(req.From, MsgPushConnectRsp, rsp)
 		return ErrVersionNotCompatible
 	}
 	// verify totp token or token
 	t := totp.TOTP{Step: totp.RelayTOTPStep}
-	if t.Verify(req.Token, pn.config.Token, time.Now().Unix()-pn.dt/int64(time.Second)) { // localTs may behind, auto adjust ts
-		gLog.Printf(LvINFO, "Access Granted\n")
+	if t.Verify(req.Token, gConf.Network.Token, time.Now().Unix()-GNetwork.dt/int64(time.Second)) { // localTs may behind, auto adjust ts
+		gLog.Printf(LvINFO, "Access Granted")
 		config := AppConfig{}
 		config.peerNatType = req.NatType
 		config.peerConeNatPort = req.ConeNatPort
@@ -185,52 +260,74 @@ func handleConnectReq(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 		config.hasUPNPorNATPMP = req.HasUPNPorNATPMP
 		config.linkMode = req.LinkMode
 		config.isUnderlayServer = req.IsUnderlayServer
+		config.UnderlayProtocol = req.UnderlayProtocol
 		// share relay node will limit bandwidth
-		if req.Token != pn.config.Token {
-			gLog.Printf(LvINFO, "set share bandwidth %d mbps", pn.config.ShareBandwidth)
-			config.shareBandwidth = pn.config.ShareBandwidth
+		if req.Token != gConf.Network.Token {
+			gLog.Printf(LvINFO, "set share bandwidth %d mbps", gConf.Network.ShareBandwidth)
+			config.shareBandwidth = gConf.Network.ShareBandwidth
 		}
-		// go pn.AddTunnel(config, req.ID)
-		go pn.addDirectTunnel(config, req.ID)
+		// go GNetwork.AddTunnel(config, req.ID)
+		go func() {
+			GNetwork.addDirectTunnel(config, req.ID)
+		}()
 		return nil
 	}
 	gLog.Println(LvERROR, "Access Denied:", req.From)
 	rsp := PushConnectRsp{
 		Error:  1,
-		Detail: fmt.Sprintf("connect to %s error: Access Denied", pn.config.Node),
+		Detail: fmt.Sprintf("connect to %s error: Access Denied", gConf.Network.Node),
 		To:     req.From,
-		From:   pn.config.Node,
+		From:   gConf.Network.Node,
 	}
-	return pn.push(req.From, MsgPushConnectRsp, rsp)
+	return GNetwork.push(req.From, MsgPushConnectRsp, rsp)
 }
 
-func handleReportApps(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
+func handleReportApps() (err error) {
 	gLog.Println(LvINFO, "MsgPushReportApps")
 	req := ReportApps{}
 	gConf.mtx.Lock()
 	defer gConf.mtx.Unlock()
+
 	for _, config := range gConf.Apps {
 		appActive := 0
 		relayNode := ""
+		specRelayNode := ""
 		relayMode := ""
 		linkMode := LinkModeUDPPunch
-		i, ok := pn.apps.Load(config.ID())
+		var connectTime string
+		var retryTime string
+		var app *p2pApp
+		i, ok := GNetwork.apps.Load(config.ID())
 		if ok {
-			app := i.(*p2pApp)
+			app = i.(*p2pApp)
 			if app.isActive() {
 				appActive = 1
 			}
+			if app.config.SrcPort == 0 { // memapp
+				continue
+			}
+			specRelayNode = app.config.RelayNode
+			if !app.isDirect() { // TODO: should always report relay node for app edit
 				relayNode = app.relayNode
 				relayMode = app.relayMode
-			linkMode = app.tunnel.linkModeWeb
+			}
+
+			if app.Tunnel() != nil {
+				linkMode = app.Tunnel().linkModeWeb
+			}
+			retryTime = app.RetryTime().Local().Format("2006-01-02T15:04:05-0700")
+			connectTime = app.ConnectTime().Local().Format("2006-01-02T15:04:05-0700")
+
 		}
 		appInfo := AppInfo{
 			AppName:       config.AppName,
 			Error:         config.errMsg,
 			Protocol:      config.Protocol,
+			PunchPriority: config.PunchPriority,
 			Whitelist:     config.Whitelist,
 			SrcPort:       config.SrcPort,
 			RelayNode:     relayNode,
+			SpecRelayNode: specRelayNode,
 			RelayMode:     relayMode,
 			LinkMode:      linkMode,
 			PeerNode:      config.PeerNode,
@@ -239,17 +336,77 @@ func handleReportApps(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 			PeerUser:      config.PeerUser,
 			PeerIP:        config.peerIP,
 			PeerNatType:   config.peerNatType,
-			RetryTime:   config.retryTime.Local().Format("2006-01-02T15:04:05-0700"),
-			ConnectTime: config.connectTime.Local().Format("2006-01-02T15:04:05-0700"),
+			RetryTime:     retryTime,
+			ConnectTime:   connectTime,
 			IsActive:      appActive,
 			Enabled:       config.Enabled,
 		}
 		req.Apps = append(req.Apps, appInfo)
 	}
-	return pn.write(MsgReport, MsgReportApps, &req)
+	return GNetwork.write(MsgReport, MsgReportApps, &req)
+
 }
 
-func handleLog(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
+func handleReportMemApps() (err error) {
+	gLog.Println(LvINFO, "handleReportMemApps")
+	req := ReportApps{}
+	gConf.mtx.Lock()
+	defer gConf.mtx.Unlock()
+	GNetwork.sdwan.sysRoute.Range(func(key, value interface{}) bool {
+		node := value.(*sdwanNode)
+		appActive := 0
+		relayMode := ""
+		var connectTime string
+		var retryTime string
+
+		i, ok := GNetwork.apps.Load(node.id)
+		var app *p2pApp
+		if ok {
+			app = i.(*p2pApp)
+			if app.isActive() {
+				appActive = 1
+			}
+			if !app.isDirect() {
+				relayMode = app.relayMode
+			}
+			retryTime = app.RetryTime().Local().Format("2006-01-02T15:04:05-0700")
+			connectTime = app.ConnectTime().Local().Format("2006-01-02T15:04:05-0700")
+		}
+		appInfo := AppInfo{
+			RelayMode: relayMode,
+			PeerNode:  node.name,
+			IsActive:  appActive,
+			Enabled:   1,
+		}
+		if app != nil {
+			appInfo.AppName = app.config.AppName
+			appInfo.Error = app.config.errMsg
+			appInfo.Protocol = app.config.Protocol
+			appInfo.Whitelist = app.config.Whitelist
+			appInfo.SrcPort = app.config.SrcPort
+			if !app.isDirect() {
+				appInfo.RelayNode = app.relayNode
+			}
+
+			if app.Tunnel() != nil {
+				appInfo.LinkMode = app.Tunnel().linkModeWeb
+			}
+			appInfo.DstHost = app.config.DstHost
+			appInfo.DstPort = app.config.DstPort
+			appInfo.PeerUser = app.config.PeerUser
+			appInfo.PeerIP = app.config.peerIP
+			appInfo.PeerNatType = app.config.peerNatType
+			appInfo.RetryTime = retryTime
+			appInfo.ConnectTime = connectTime
+		}
+		req.Apps = append(req.Apps, appInfo)
+		return true
+	})
+	gLog.Println(LvDEBUG, "handleReportMemApps res:", prettyJson(req))
+	return GNetwork.write(MsgReport, MsgReportMemApps, &req)
+}
+
+func handleLog(msg []byte) (err error) {
 	gLog.Println(LvDEBUG, "MsgPushReportLog")
 	const defaultLen = 1024 * 128
 	const maxLen = 1024 * 1024
@@ -260,6 +417,8 @@ func handleLog(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 	}
 	if req.FileName == "" {
 		req.FileName = "openp2p.log"
+	} else {
+		req.FileName = sanitizeFileName(req.FileName)
 	}
 	f, err := os.Open(filepath.Join("log", req.FileName))
 	if err != nil {
@@ -294,5 +453,30 @@ func handleLog(pn *P2PNetwork, subType uint16, msg []byte) (err error) {
 	rsp.FileName = req.FileName
 	rsp.Total = fi.Size()
 	rsp.Len = req.Len
-	return pn.write(MsgReport, MsgPushReportLog, &rsp)
+	return GNetwork.write(MsgReport, MsgPushReportLog, &rsp)
+}
+
+func handleReportGoroutine() (err error) {
+	gLog.Println(LvDEBUG, "handleReportGoroutine")
+	buf := make([]byte, 1024*128)
+	stackLen := runtime.Stack(buf, true)
+	return GNetwork.write(MsgReport, MsgPushReportLog, string(buf[:stackLen]))
+}
+
+func handleCheckRemoteService(msg []byte) (err error) {
+	gLog.Println(LvDEBUG, "handleCheckRemoteService")
+	req := CheckRemoteService{}
+	if err = json.Unmarshal(msg[openP2PHeaderSize:], &req); err != nil {
+		gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(req), err, string(msg[openP2PHeaderSize:]))
+		return err
+	}
+	rsp := PushRsp{Error: 0}
+	conn, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.Host, req.Port), time.Second*3)
+	if err != nil {
+		rsp.Error = 1
+		rsp.Detail = ErrRemoteServiceUnable.Error()
+	} else {
+		conn.Close()
+	}
+	return GNetwork.write(MsgReport, MsgReportResponse, rsp)
 }

core/holepunch.go

@@ -3,6 +3,7 @@ package openp2p
 import (
 	"bytes"
 	"encoding/binary"
+	"encoding/json"
 	"fmt"
 	"math/rand"
 	"net"
@@ -10,41 +11,45 @@ import (
 )
 
 func handshakeC2C(t *P2PTunnel) (err error) {
-	gLog.Printf(LvDEBUG, "handshakeC2C %s:%d:%d to %s:%d", t.pn.config.Node, t.coneLocalPort, t.coneNatPort, t.config.peerIP, t.config.peerConeNatPort)
+	gLog.Printf(LvDEBUG, "handshakeC2C %s:%d:%d to %s:%d", gConf.Network.Node, t.coneLocalPort, t.coneNatPort, t.config.peerIP, t.config.peerConeNatPort)
 	defer gLog.Printf(LvDEBUG, "handshakeC2C end")
-	conn, err := net.ListenUDP("udp", t.la)
+	conn, err := net.ListenUDP("udp", t.localHoleAddr)
 	if err != nil {
 		return err
 	}
 	defer conn.Close()
-	_, err = UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshake, P2PHandshakeReq{ID: t.id})
+	_, err = UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshake, P2PHandshakeReq{ID: t.id})
 	if err != nil {
 		gLog.Println(LvDEBUG, "handshakeC2C write MsgPunchHandshake error:", err)
 		return err
 	}
-	ra, head, _, _, err := UDPRead(conn, HandshakeTimeout)
-	if err != nil {
-		time.Sleep(time.Millisecond * 200)
-		gLog.Println(LvDEBUG, err, ", return this error when ip was not reachable, retry read")
-		ra, head, _, _, err = UDPRead(conn, HandshakeTimeout)
+	ra, head, buff, _, err := UDPRead(conn, HandshakeTimeout)
 	if err != nil {
 		gLog.Println(LvDEBUG, "handshakeC2C read MsgPunchHandshake error:", err)
 		return err
 	}
+	t.remoteHoleAddr, _ = net.ResolveUDPAddr("udp", ra.String())
+	var tunnelID uint64
+	if len(buff) > openP2PHeaderSize {
+		req := P2PHandshakeReq{}
+		if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+			tunnelID = req.ID
 		}
-	t.ra, _ = net.ResolveUDPAddr("udp", ra.String())
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake {
+	} else { // compatible with old version
+		tunnelID = t.id
+	}
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
 		gLog.Printf(LvDEBUG, "read %d handshake ", t.id)
-		UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 		_, head, _, _, err = UDPRead(conn, HandshakeTimeout)
 		if err != nil {
 			gLog.Println(LvDEBUG, "handshakeC2C write MsgPunchHandshakeAck error", err)
 			return err
 		}
 	}
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
 		gLog.Printf(LvDEBUG, "read %d handshake ack ", t.id)
-		_, err = UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		_, err = UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 		if err != nil {
 			gLog.Println(LvDEBUG, "handshakeC2C write MsgPunchHandshakeAck error", err)
 			return err
@@ -57,9 +62,15 @@ func handshakeC2C(t *P2PTunnel) (err error) {
 func handshakeC2S(t *P2PTunnel) error {
 	gLog.Printf(LvDEBUG, "handshakeC2S start")
 	defer gLog.Printf(LvDEBUG, "handshakeC2S end")
+	if !buildTunnelMtx.TryLock() {
+		// time.Sleep(time.Second * 3)
+		return ErrBuildTunnelBusy
+	}
+	defer buildTunnelMtx.Unlock()
+	startTime := time.Now()
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 	randPorts := r.Perm(65532)
-	conn, err := net.ListenUDP("udp", t.la)
+	conn, err := net.ListenUDP("udp", t.localHoleAddr)
 	if err != nil {
 		return err
 	}
@@ -68,7 +79,6 @@ func handshakeC2S(t *P2PTunnel) error {
 	go func() error {
 		gLog.Printf(LvDEBUG, "send symmetric handshake to %s from %d:%d start", t.config.peerIP, t.coneLocalPort, t.coneNatPort)
 		for i := 0; i < SymmetricHandshakeNum; i++ {
-			// TODO: auto calc cost time
 			// time.Sleep(SymmetricHandshakeInterval)
 			dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, randPorts[i]+2))
 			if err != nil {
@@ -89,54 +99,77 @@ func handshakeC2S(t *P2PTunnel) error {
 		return err
 	}
 	// read response of the punching hole ok port
-	result := make([]byte, 1024)
-	_, dst, err := conn.ReadFrom(result)
+	buff := make([]byte, 1024)
+	_, dst, err := conn.ReadFrom(buff)
 	if err != nil {
 		gLog.Println(LvERROR, "handshakeC2S wait timeout")
 		return err
 	}
 	head := &openP2PHeader{}
-	err = binary.Read(bytes.NewReader(result[:openP2PHeaderSize]), binary.LittleEndian, head)
+	err = binary.Read(bytes.NewReader(buff[:openP2PHeaderSize]), binary.LittleEndian, head)
 	if err != nil {
 		gLog.Println(LvERROR, "parse p2pheader error:", err)
 		return err
 	}
-	t.ra, _ = net.ResolveUDPAddr("udp", dst.String())
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake {
+	t.remoteHoleAddr, _ = net.ResolveUDPAddr("udp", dst.String())
+	var tunnelID uint64
+	if len(buff) > openP2PHeaderSize {
+		req := P2PHandshakeReq{}
+		if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+			tunnelID = req.ID
+		}
+	} else { // compatible with old version
+		tunnelID = t.id
+	}
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
 		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ", t.id)
-		UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 		for {
-			_, head, _, _, err = UDPRead(conn, HandshakeTimeout)
+			_, head, buff, _, err = UDPRead(conn, HandshakeTimeout)
 			if err != nil {
 				gLog.Println(LvDEBUG, "handshakeC2S handshake error")
 				return err
 			}
+			var tunnelID uint64
+			if len(buff) > openP2PHeaderSize {
+				req := P2PHandshakeReq{}
+				if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+					tunnelID = req.ID
+				}
+			} else { // compatible with old version
+				tunnelID = t.id
+			}
 			// waiting ack
-			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
 				break
 			}
 		}
 	}
 	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
-		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ack %s", t.id, t.ra.String())
-		_, err = UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ack %s", t.id, t.remoteHoleAddr.String())
+		_, err = UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 		return err
 	} else {
 		gLog.Println(LvDEBUG, "handshakeS2C read msg but not MsgPunchHandshakeAck")
 	}
-	gLog.Printf(LvINFO, "handshakeC2S ok")
+	gLog.Printf(LvINFO, "handshakeC2S ok. cost %d ms", time.Since(startTime)/time.Millisecond)
 	return nil
 }
 
 func handshakeS2C(t *P2PTunnel) error {
 	gLog.Printf(LvDEBUG, "handshakeS2C start")
 	defer gLog.Printf(LvDEBUG, "handshakeS2C end")
+	if !buildTunnelMtx.TryLock() {
+		// time.Sleep(time.Second * 3)
+		return ErrBuildTunnelBusy
+	}
+	defer buildTunnelMtx.Unlock()
+	startTime := time.Now()
 	gotCh := make(chan *net.UDPAddr, 5)
 	// sequencely udp send handshake, do not parallel send
 	gLog.Printf(LvDEBUG, "send symmetric handshake to %s:%d start", t.config.peerIP, t.config.peerConeNatPort)
 	gotIt := false
 	for i := 0; i < SymmetricHandshakeNum; i++ {
-		// TODO: auto calc cost time
 		// time.Sleep(SymmetricHandshakeInterval)
 		go func(t *P2PTunnel) error {
 			conn, err := net.ListenUDP("udp", nil) // TODO: system allocated port really random?
@@ -145,8 +178,8 @@ func handshakeS2C(t *P2PTunnel) error {
 				return err
 			}
 			defer conn.Close()
-			UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshake, P2PHandshakeReq{ID: t.id})
-			_, head, _, _, err := UDPRead(conn, HandshakeTimeout)
+			UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshake, P2PHandshakeReq{ID: t.id})
+			_, head, buff, _, err := UDPRead(conn, HandshakeTimeout)
 			if err != nil {
 				// gLog.Println(LevelDEBUG, "one of the handshake error:", err)
 				return err
@@ -154,18 +187,35 @@ func handshakeS2C(t *P2PTunnel) error {
 			if gotIt {
 				return nil
 			}
+			var tunnelID uint64
+			if len(buff) >= openP2PHeaderSize+8 {
+				req := P2PHandshakeReq{}
+				if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+					tunnelID = req.ID
+				}
+			} else { // compatible with old version
+				tunnelID = t.id
+			}
 
-			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake {
+			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
 				gLog.Printf(LvDEBUG, "handshakeS2C read %d handshake ", t.id)
-				UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
-				// may read sereral MsgPunchHandshake
+				UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+				// may read several MsgPunchHandshake
 				for {
-					_, head, _, _, err = UDPRead(conn, HandshakeTimeout)
+					_, head, buff, _, err = UDPRead(conn, HandshakeTimeout)
 					if err != nil {
 						gLog.Println(LvDEBUG, "handshakeS2C handshake error")
 						return err
 					}
-					if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+					if len(buff) > openP2PHeaderSize {
+						req := P2PHandshakeReq{}
+						if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+							tunnelID = req.ID
+						}
+					} else { // compatible with old version
+						tunnelID = t.id
+					}
+					if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
 						break
 					} else {
 						gLog.Println(LvDEBUG, "handshakeS2C read msg but not MsgPunchHandshakeAck")
@@ -174,7 +224,7 @@ func handshakeS2C(t *P2PTunnel) error {
 			}
 			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
 				gLog.Printf(LvDEBUG, "handshakeS2C read %d handshake ack %s", t.id, conn.LocalAddr().String())
-				UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+				UDPWrite(conn, t.remoteHoleAddr, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 				gotIt = true
 				la, _ := net.ResolveUDPAddr("udp", conn.LocalAddr().String())
 				gotCh <- la
@@ -186,18 +236,18 @@ func handshakeS2C(t *P2PTunnel) error {
 		}(t)
 	}
 	gLog.Printf(LvDEBUG, "send symmetric handshake end")
-	if compareVersion(t.config.peerVersion, SymmetricSimultaneouslySendVersion) == LESS { // compatible with old client
+	if compareVersion(t.config.peerVersion, SymmetricSimultaneouslySendVersion) < 0 { // compatible with old client
 		gLog.Println(LvDEBUG, "handshakeS2C ready, notify peer connect")
-		t.pn.push(t.config.PeerNode, MsgPushHandshakeStart, TunnelMsg{ID: t.id})
+		GNetwork.push(t.config.PeerNode, MsgPushHandshakeStart, TunnelMsg{ID: t.id})
 	}
 
 	select {
 	case <-time.After(HandshakeTimeout):
-		return fmt.Errorf("wait handshake failed")
+		return fmt.Errorf("wait handshake timeout")
 	case la := <-gotCh:
-		t.la = la
+		t.localHoleAddr = la
 		gLog.Println(LvDEBUG, "symmetric handshake ok", la)
-		gLog.Printf(LvINFO, "handshakeS2C ok")
+		gLog.Printf(LvINFO, "handshakeS2C ok. cost %dms", time.Since(startTime)/time.Millisecond)
 	}
 	return nil
 }

core/install.go

@@ -10,11 +10,6 @@ import (
 	"time"
 )
 
-// examples:
-// listen:
-// ./openp2p install -node hhd1207-222 -token YOUR-TOKEN -sharebandwidth 0
-// listen and build p2papp:
-// ./openp2p install -node hhd1207-222 -token YOUR-TOKEN -sharebandwidth 0 -peernode hhdhome-n1 -dstip 127.0.0.1 -dstport 50022 -protocol tcp -srcport 22
 func install() {
 	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
 	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email openp2p.cn@gmail.com")
@@ -35,7 +30,7 @@ func install() {
 
 	uninstall()
 	// save config file
-	parseParams("install")
+	parseParams("install", "")
 	targetPath := filepath.Join(defaultInstallPath, defaultBinName)
 	d := daemon{}
 	// copy files

core/iptables.go

@@ -0,0 +1,74 @@
+package openp2p
+
+import (
+	"log"
+	"os/exec"
+	"runtime"
+)
+
+func allowTunForward() {
+	if runtime.GOOS != "linux" { // only support Linux
+		return
+	}
+	exec.Command("sh", "-c", `iptables -t filter -D FORWARD -i optun -j ACCEPT`).Run()
+	exec.Command("sh", "-c", `iptables -t filter -D FORWARD -o optun -j ACCEPT`).Run()
+	err := exec.Command("sh", "-c", `iptables -t filter -I FORWARD -i optun -j ACCEPT`).Run()
+	if err != nil {
+		log.Println("allow foward in error:", err)
+	}
+	err = exec.Command("sh", "-c", `iptables -t filter -I FORWARD -o optun -j ACCEPT`).Run()
+	if err != nil {
+		log.Println("allow foward out error:", err)
+	}
+}
+
+func clearSNATRule() {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	execCommand("iptables", true, "-t", "nat", "-D", "POSTROUTING", "-j", "OPSDWAN")
+	execCommand("iptables", true, "-t", "nat", "-F", "OPSDWAN")
+	execCommand("iptables", true, "-t", "nat", "-X", "OPSDWAN")
+}
+
+func initSNATRule(localNet string) {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	clearSNATRule()
+
+	err := execCommand("iptables", true, "-t", "nat", "-N", "OPSDWAN")
+	if err != nil {
+		log.Println("iptables new sdwan chain error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "POSTROUTING", "-j", "OPSDWAN")
+	if err != nil {
+		log.Println("iptables append postrouting error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN",
+		"-o", "optun", "!", "-s", localNet, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("add optun snat error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN", "!", "-o", "optun",
+		"-s", localNet, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("add optun snat error:", err)
+		return
+	}
+}
+
+func addSNATRule(target string) {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	err := execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN", "!", "-o", "optun",
+		"-s", target, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("iptables add optun snat error:", err)
+		return
+	}
+}

core/iptree.go

@@ -17,9 +17,18 @@ type IPTree struct {
 	tree    *avltree.Tree
 	treeMtx sync.RWMutex
 }
+type IPTreeValue struct {
+	maxIP uint32
+	v     interface{}
+}
+
+// TODO: deal interset
+func (iptree *IPTree) DelIntIP(minIP uint32, maxIP uint32) {
+	iptree.tree.Remove(minIP)
+}
 
 // add 120k cost 0.5s
-func (iptree *IPTree) AddIntIP(minIP uint32, maxIP uint32) bool {
+func (iptree *IPTree) AddIntIP(minIP uint32, maxIP uint32, v interface{}) bool {
 	if minIP > maxIP {
 		return false
 	}
@@ -32,15 +41,15 @@ func (iptree *IPTree) AddIntIP(minIP uint32, maxIP uint32) bool {
 		if cur == nil {
 			break
 		}
-		curMaxIP := cur.Value.(uint32)
+		tv := cur.Value.(*IPTreeValue)
 		curMinIP := cur.Key.(uint32)
 
 		// newNode all in existNode, treat as inserted.
-		if newMinIP >= curMinIP && newMaxIP <= curMaxIP {
+		if newMinIP >= curMinIP && newMaxIP <= tv.maxIP {
 			return true
 		}
 		// has no interset
-		if newMinIP > curMaxIP {
+		if newMinIP > tv.maxIP {
 			cur = cur.Children[1]
 			continue
 		}
@@ -53,49 +62,64 @@ func (iptree *IPTree) AddIntIP(minIP uint32, maxIP uint32) bool {
 		if curMinIP < newMinIP {
 			newMinIP = curMinIP
 		}
-		if curMaxIP > newMaxIP {
-			newMaxIP = curMaxIP
+		if tv.maxIP > newMaxIP {
+			newMaxIP = tv.maxIP
 		}
 		cur = iptree.tree.Root
 	}
 	//  put in the tree
-	iptree.tree.Put(newMinIP, newMaxIP)
+	iptree.tree.Put(newMinIP, &IPTreeValue{newMaxIP, v})
 	return true
 }
 
-func (iptree *IPTree) Add(minIPStr string, maxIPStr string) bool {
+func (iptree *IPTree) Add(minIPStr string, maxIPStr string, v interface{}) bool {
 	var minIP, maxIP uint32
 	binary.Read(bytes.NewBuffer(net.ParseIP(minIPStr).To4()), binary.BigEndian, &minIP)
 	binary.Read(bytes.NewBuffer(net.ParseIP(maxIPStr).To4()), binary.BigEndian, &maxIP)
-	return iptree.AddIntIP(minIP, maxIP)
+	return iptree.AddIntIP(minIP, maxIP, v)
+}
+
+func (iptree *IPTree) Del(minIPStr string, maxIPStr string) {
+	var minIP, maxIP uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP(minIPStr).To4()), binary.BigEndian, &minIP)
+	binary.Read(bytes.NewBuffer(net.ParseIP(maxIPStr).To4()), binary.BigEndian, &maxIP)
+	iptree.DelIntIP(minIP, maxIP)
 }
 
 func (iptree *IPTree) Contains(ipStr string) bool {
 	var ip uint32
 	binary.Read(bytes.NewBuffer(net.ParseIP(ipStr).To4()), binary.BigEndian, &ip)
-	return iptree.ContainsInt(ip)
+	_, ok := iptree.Load(ip)
+	return ok
 }
 
-func (iptree *IPTree) ContainsInt(ip uint32) bool {
+func IsLocalhost(ipStr string) bool {
+	if ipStr == "localhost" || ipStr == "127.0.0.1" || ipStr == "::1" {
+		return true
+	}
+	return false
+}
+
+func (iptree *IPTree) Load(ip uint32) (interface{}, bool) {
 	iptree.treeMtx.RLock()
 	defer iptree.treeMtx.RUnlock()
 	if iptree.tree == nil {
-		return false
+		return nil, false
 	}
 	n := iptree.tree.Root
 	for n != nil {
-		curMaxIP := n.Value.(uint32)
+		tv := n.Value.(*IPTreeValue)
 		curMinIP := n.Key.(uint32)
 		switch {
-		case ip >= curMinIP && ip <= curMaxIP: // hit
-			return true
+		case ip >= curMinIP && ip <= tv.maxIP: // hit
+			return tv.v, true
 		case ip < curMinIP:
 			n = n.Children[0]
 		default:
 			n = n.Children[1]
 		}
 	}
-	return false
+	return nil, false
 }
 
 func (iptree *IPTree) Size() int {
@@ -135,12 +159,12 @@ func NewIPTree(ips string) *IPTree {
 			}
 			minIP := ipNet.IP.Mask(ipNet.Mask).String()
 			maxIP := calculateMaxIP(ipNet).String()
-			iptree.Add(minIP, maxIP)
+			iptree.Add(minIP, maxIP, nil)
 		} else if strings.Contains(ip, "-") { // x.x.x.x-y.y.y.y
 			minAndMax := strings.Split(ip, "-")
-			iptree.Add(minAndMax[0], minAndMax[1])
+			iptree.Add(minAndMax[0], minAndMax[1], nil)
 		} else { // single ip
-			iptree.Add(ip, ip)
+			iptree.Add(ip, ip, nil)
 		}
 	}
 	return iptree

core/iptree_test.go

@@ -5,6 +5,7 @@ import (
 	"encoding/binary"
 	"net"
 	"testing"
+	"time"
 )
 
 func wrapTestContains(t *testing.T, iptree *IPTree, ip string, result bool) {
@@ -40,14 +41,14 @@ func TestAllInputFormat(t *testing.T) {
 
 func TestSingleIP(t *testing.T) {
 	iptree := NewIPTree("")
-	iptree.Add("219.137.185.70", "219.137.185.70")
+	iptree.Add("219.137.185.70", "219.137.185.70", nil)
 	wrapTestContains(t, iptree, "219.137.185.70", true)
 	wrapTestContains(t, iptree, "219.137.185.71", false)
 }
 
 func TestWrongSegment(t *testing.T) {
 	iptree := NewIPTree("")
-	inserted := iptree.Add("87.251.75.0", "82.251.75.255")
+	inserted := iptree.Add("87.251.75.0", "82.251.75.255", nil)
 	if inserted {
 		t.Errorf("TestWrongSegment failed\n")
 	}
@@ -56,20 +57,20 @@ func TestWrongSegment(t *testing.T) {
 func TestSegment2(t *testing.T) {
 	iptree := NewIPTree("")
 	iptree.Clear()
-	iptree.Add("10.1.5.50", "10.1.5.100")
-	iptree.Add("10.1.1.50", "10.1.1.100")
-	iptree.Add("10.1.2.50", "10.1.2.100")
-	iptree.Add("10.1.6.50", "10.1.6.100")
-	iptree.Add("10.1.7.50", "10.1.7.100")
-	iptree.Add("10.1.3.50", "10.1.3.100")
-	iptree.Add("10.1.1.1", "10.1.1.10")    // no interset
-	iptree.Add("10.1.1.200", "10.1.1.250") // no interset
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
 	iptree.Print()
 
-	iptree.Add("10.1.1.80", "10.1.1.90") // all in
-	iptree.Add("10.1.1.40", "10.1.1.60") // interset
+	iptree.Add("10.1.1.80", "10.1.1.90", nil) // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil) // interset
 	iptree.Print()
-	iptree.Add("10.1.1.90", "10.1.1.110") // interset
+	iptree.Add("10.1.1.90", "10.1.1.110", nil) // interset
 	iptree.Print()
 	t.Logf("ipTree size:%d\n", iptree.Size())
 	wrapTestContains(t, iptree, "10.1.1.40", true)
@@ -86,7 +87,7 @@ func TestSegment2(t *testing.T) {
 	wrapTestContains(t, iptree, "10.1.100.30", false)
 	wrapTestContains(t, iptree, "10.1.200.30", false)
 
-	iptree.Add("10.0.0.0", "10.255.255.255") // will merge all segment
+	iptree.Add("10.0.0.0", "10.255.255.255", nil) // will merge all segment
 	iptree.Print()
 	if iptree.Size() != 1 {
 		t.Errorf("merge ip segment error\n")
@@ -97,17 +98,17 @@ func TestSegment2(t *testing.T) {
 func BenchmarkBuildipTree20k(t *testing.B) {
 	iptree := NewIPTree("")
 	iptree.Clear()
-	iptree.Add("10.1.5.50", "10.1.5.100")
-	iptree.Add("10.1.1.50", "10.1.1.100")
-	iptree.Add("10.1.2.50", "10.1.2.100")
-	iptree.Add("10.1.6.50", "10.1.6.100")
-	iptree.Add("10.1.7.50", "10.1.7.100")
-	iptree.Add("10.1.3.50", "10.1.3.100")
-	iptree.Add("10.1.1.1", "10.1.1.10")    // no interset
-	iptree.Add("10.1.1.200", "10.1.1.250") // no interset
-	iptree.Add("10.1.1.80", "10.1.1.90")   // all in
-	iptree.Add("10.1.1.40", "10.1.1.60")   // interset
-	iptree.Add("10.1.1.90", "10.1.1.110")  // interset
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
+	iptree.Add("10.1.1.80", "10.1.1.90", nil)   // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil)   // interset
+	iptree.Add("10.1.1.90", "10.1.1.110", nil)  // interset
 	var minIP uint32
 	binary.Read(bytes.NewBuffer(net.ParseIP("10.1.1.1").To4()), binary.BigEndian, &minIP)
 
@@ -115,57 +116,59 @@ func BenchmarkBuildipTree20k(t *testing.B) {
 	nodeNum := uint32(10000 * 1)
 	gap := uint32(10)
 	for i := minIP; i < minIP+nodeNum*gap; i += gap {
-		iptree.AddIntIP(i, i)
+		iptree.AddIntIP(i, i, nil)
 		// t.Logf("ipTree size:%d\n", iptree.Size())
 	}
 	binary.Read(bytes.NewBuffer(net.ParseIP("100.1.1.1").To4()), binary.BigEndian, &minIP)
 	// insert 100k block ip segment
 	for i := minIP; i < minIP+nodeNum*gap; i += gap {
-		iptree.AddIntIP(i, i+5)
+		iptree.AddIntIP(i, i+5, nil)
 	}
 	t.Logf("ipTree size:%d\n", iptree.Size())
 	iptree.Clear()
 	t.Logf("clear. ipTree size:%d\n", iptree.Size())
 }
 func BenchmarkQuery(t *testing.B) {
+	ts := time.Now()
 	iptree := NewIPTree("")
 	iptree.Clear()
-	iptree.Add("10.1.5.50", "10.1.5.100")
-	iptree.Add("10.1.1.50", "10.1.1.100")
-	iptree.Add("10.1.2.50", "10.1.2.100")
-	iptree.Add("10.1.6.50", "10.1.6.100")
-	iptree.Add("10.1.7.50", "10.1.7.100")
-	iptree.Add("10.1.3.50", "10.1.3.100")
-	iptree.Add("10.1.1.1", "10.1.1.10")    // no interset
-	iptree.Add("10.1.1.200", "10.1.1.250") // no interset
-	iptree.Add("10.1.1.80", "10.1.1.90")   // all in
-	iptree.Add("10.1.1.40", "10.1.1.60")   // interset
-	iptree.Add("10.1.1.90", "10.1.1.110")  // interset
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
+	iptree.Add("10.1.1.80", "10.1.1.90", nil)   // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil)   // interset
+	iptree.Add("10.1.1.90", "10.1.1.110", nil)  // interset
 	var minIP uint32
 	binary.Read(bytes.NewBuffer(net.ParseIP("10.1.1.1").To4()), binary.BigEndian, &minIP)
 
 	// insert 10k block ip single
-	nodeNum := uint32(10000 * 100)
+	nodeNum := uint32(10000 * 1000)
 	gap := uint32(10)
 	for i := minIP; i < minIP+nodeNum*gap; i += gap {
-		iptree.AddIntIP(i, i)
+		iptree.AddIntIP(i, i, nil)
 		// t.Logf("ipTree size:%d\n", iptree.Size())
 	}
 	binary.Read(bytes.NewBuffer(net.ParseIP("100.1.1.1").To4()), binary.BigEndian, &minIP)
 	// insert 100k block ip segment
 	for i := minIP; i < minIP+nodeNum*gap; i += gap {
-		iptree.AddIntIP(i, i+5)
+		iptree.AddIntIP(i, i+5, nil)
 	}
-	t.Logf("ipTree size:%d\n", iptree.Size())
-	t.ResetTimer()
+	t.Logf("ipTree size:%d cost:%dms\n", iptree.Size(), time.Since(ts)/time.Millisecond)
+	ts = time.Now()
+	// t.ResetTimer()
 	queryNum := 100 * 10000
 	for i := 0; i < queryNum; i++ {
-		iptree.ContainsInt(minIP + uint32(i))
+		iptree.Load(minIP + uint32(i))
 		wrapBenchmarkContains(t, iptree, "10.1.5.55", true)
 		wrapBenchmarkContains(t, iptree, "10.1.1.1", true)
 		wrapBenchmarkContains(t, iptree, "10.1.5.200", false)
 		wrapBenchmarkContains(t, iptree, "200.1.1.1", false)
 	}
-	t.Logf("query list:%d\n", queryNum*4)
+	t.Logf("query num:%d cost:%dms\n", queryNum*4, time.Since(ts)/time.Millisecond)
 
 }

core/log.go

@@ -13,6 +13,7 @@ type LogLevel int
 var gLog *logger
 
 const (
+	LvDev   LogLevel = -1
 	LvDEBUG LogLevel = iota
 	LvINFO
 	LvWARN
@@ -32,12 +33,13 @@ func init() {
 	loglevel[LvINFO] = "INFO"
 	loglevel[LvWARN] = "WARN"
 	loglevel[LvERROR] = "ERROR"
+	loglevel[LvDev] = "Dev"
 
 }
 
 const (
-	LogFile = 1 << iota
-	LogConsole
+	LogFile    = 1
+	LogConsole = 1 << 1
 )
 
 type logger struct {
@@ -92,6 +94,13 @@ func (l *logger) setLevel(level LogLevel) {
 	defer l.mtx.Unlock()
 	l.level = level
 }
+
+func (l *logger) setMaxSize(size int64) {
+	l.mtx.Lock()
+	defer l.mtx.Unlock()
+	l.maxLogSize = size
+}
+
 func (l *logger) setMode(mode int) {
 	l.mtx.Lock()
 	defer l.mtx.Unlock()
@@ -139,10 +148,10 @@ func (l *logger) Printf(level LogLevel, format string, params ...interface{}) {
 	}
 	pidAndLevel := []interface{}{l.pid, loglevel[level]}
 	params = append(pidAndLevel, params...)
-	if l.mode & LogFile != 0 {
+	if l.mode&LogFile != 0 {
 		l.loggers[0].Printf("%d %s "+format+l.lineEnding, params...)
 	}
-	if l.mode & LogConsole != 0 {
+	if l.mode&LogConsole != 0 {
 		l.stdLogger.Printf("%d %s "+format+l.lineEnding, params...)
 	}
 }
@@ -156,10 +165,10 @@ func (l *logger) Println(level LogLevel, params ...interface{}) {
 	pidAndLevel := []interface{}{l.pid, " ", loglevel[level], " "}
 	params = append(pidAndLevel, params...)
 	params = append(params, l.lineEnding)
-	if l.mode & LogFile != 0 {
+	if l.mode&LogFile != 0 {
 		l.loggers[0].Print(params...)
 	}
-	if l.mode & LogConsole != 0 {
+	if l.mode&LogConsole != 0 {
 		l.stdLogger.Print(params...)
 	}
 }

core/nat.go

@@ -66,7 +66,7 @@ func natTest(serverHost string, serverPort int, localPort int) (publicIP string,
 	}
 
 	// The connection can write data to the desired address.
-	msg, err := newMessage(MsgNATDetect, 0, nil)
+	msg, err := newMessage(MsgNATDetect, MsgNAT, nil)
 	_, err = conn.WriteTo(msg, dst)
 	if err != nil {
 		return "", 0, err
@@ -88,29 +88,30 @@ func natTest(serverHost string, serverPort int, localPort int) (publicIP string,
 	return natRsp.IP, natRsp.Port, nil
 }
 
-func getNATType(host string, udp1 int, udp2 int) (publicIP string, NATType int, hasIPvr int, hasUPNPorNATPMP int, err error) {
+func getNATType(host string, udp1 int, udp2 int) (publicIP string, NATType int, err error) {
 	// the random local port may be used by other.
 	localPort := int(rand.Uint32()%15000 + 50000)
-	echoPort := gConf.Network.TCPPort
+
 	ip1, port1, err := natTest(host, udp1, localPort)
 	if err != nil {
-		return "", 0, 0, 0, err
+		return "", 0, err
 	}
-	hasIPv4, hasUPNPorNATPMP := publicIPTest(ip1, echoPort)
-	gLog.Printf(LvINFO, "local port:%d, nat port:%d, hasIPv4:%d, UPNP:%d", localPort, port1, hasIPv4, hasUPNPorNATPMP)
 	_, port2, err := natTest(host, udp2, localPort) // 2rd nat test not need testing publicip
 	gLog.Printf(LvDEBUG, "local port:%d  nat port:%d", localPort, port2)
 	if err != nil {
-		return "", 0, hasIPv4, hasUPNPorNATPMP, err
+		return "", 0, err
 	}
 	natType := NATSymmetric
 	if port1 == port2 {
 		natType = NATCone
 	}
-	return ip1, natType, hasIPv4, hasUPNPorNATPMP, nil
+	return ip1, natType, nil
 }
 
 func publicIPTest(publicIP string, echoPort int) (hasPublicIP int, hasUPNPorNATPMP int) {
+	if publicIP == "" || echoPort == 0 {
+		return
+	}
 	var echoConn *net.UDPConn
 	gLog.Println(LvDEBUG, "echo server start")
 	var err error
@@ -120,18 +121,6 @@ func publicIPTest(publicIP string, echoPort int) (hasPublicIP int, hasUPNPorNATP
 		return
 	}
 	defer echoConn.Close()
-	go func() {
-		// close outside for breaking the ReadFromUDP
-		// wait 30s for echo testing
-		buf := make([]byte, 1600)
-		echoConn.SetReadDeadline(time.Now().Add(time.Second * 30))
-		n, addr, err := echoConn.ReadFromUDP(buf)
-		if err != nil {
-			return
-		}
-		echoConn.WriteToUDP(buf[0:n], addr)
-		gLog.Println(LvDEBUG, "echo server end")
-	}()
 	// testing for public ip
 	for i := 0; i < 2; i++ {
 		if i == 1 {
@@ -163,17 +152,33 @@ func publicIPTest(publicIP string, echoPort int) (hasPublicIP int, hasUPNPorNATP
 			break
 		}
 		defer conn.Close()
-		dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", publicIP, echoPort))
+		dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", gConf.Network.ServerHost, gConf.Network.ServerPort))
 		if err != nil {
 			break
 		}
-		conn.WriteTo([]byte("echo"), dst)
+
+		// The connection can write data to the desired address.
+		msg, _ := newMessage(MsgNATDetect, MsgPublicIP, NatDetectReq{EchoPort: echoPort})
+		_, err = conn.WriteTo(msg, dst)
+		if err != nil {
+			continue
+		}
 		buf := make([]byte, 1600)
 
 		// wait for echo testing
-		conn.SetReadDeadline(time.Now().Add(PublicIPEchoTimeout))
-		_, _, err = conn.ReadFromUDP(buf)
-		if err == nil {
+		echoConn.SetReadDeadline(time.Now().Add(PublicIPEchoTimeout))
+		nRead, _, err := echoConn.ReadFromUDP(buf)
+		if err != nil {
+			gLog.Println(LvDEBUG, "PublicIP detect error:", err)
+			continue
+		}
+		natRsp := NatDetectRsp{}
+		err = json.Unmarshal(buf[openP2PHeaderSize:nRead], &natRsp)
+		if err != nil {
+			gLog.Println(LvDEBUG, "PublicIP detect error:", err)
+			continue
+		}
+		if natRsp.Port == echoPort {
 			if i == 1 {
 				gLog.Println(LvDEBUG, "UPNP or NAT-PMP:YES")
 				hasUPNPorNATPMP = 1

core/openp2p.go

@@ -9,12 +9,13 @@ import (
 	"time"
 )
 
+var GNetwork *P2PNetwork
+
 func Run() {
 	rand.Seed(time.Now().UnixNano())
 	baseDir := filepath.Dir(os.Args[0])
 	os.Chdir(baseDir) // for system service
 	gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFile|LogConsole)
-	// TODO: install sub command, deamon process
 	if len(os.Args) > 1 {
 		switch os.Args[1] {
 		case "version", "-v", "--version":
@@ -30,7 +31,7 @@ func Run() {
 	} else {
 		installByFilename()
 	}
-	parseParams("")
+	parseParams("", "")
 	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
 	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email openp2p.cn@gmail.com")
 
@@ -46,44 +47,67 @@ func Run() {
 	if err != nil {
 		gLog.Println(LvINFO, "setRLimit error:", err)
 	}
-	network := P2PNetworkInstance(&gConf.Network)
-	if ok := network.Connect(30000); !ok {
+	GNetwork = P2PNetworkInstance()
+	if ok := GNetwork.Connect(30000); !ok {
 		gLog.Println(LvERROR, "P2PNetwork login error")
 		return
 	}
-	gLog.Println(LvINFO, "waiting for connection...")
+	// gLog.Println(LvINFO, "waiting for connection...")
 	forever := make(chan bool)
 	<-forever
 }
 
-var network *P2PNetwork
-
 // for Android app
 // gomobile not support uint64 exported to java
+
 func RunAsModule(baseDir string, token string, bw int, logLevel int) *P2PNetwork {
 	rand.Seed(time.Now().UnixNano())
 	os.Chdir(baseDir) // for system service
-	gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFile|LogConsole)
+	gLog = NewLogger(baseDir, ProductName, LvINFO, 1024*1024, LogFile|LogConsole)
 
-	parseParams("")
+	parseParams("", "")
 
 	n, err := strconv.ParseUint(token, 10, 64)
-	if err == nil {
+	if err == nil && n > 0 {
 		gConf.setToken(n)
 	}
-	gLog.setLevel(LogLevel(logLevel))
+	if n <= 0 && gConf.Network.Token == 0 { // not input token
+		return nil
+	}
+	// gLog.setLevel(LogLevel(logLevel))
 	gConf.setShareBandwidth(bw)
 	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
 	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email openp2p.cn@gmail.com")
 	gLog.Println(LvINFO, &gConf)
 
-	network = P2PNetworkInstance(&gConf.Network)
-	if ok := network.Connect(30000); !ok {
+	GNetwork = P2PNetworkInstance()
+	if ok := GNetwork.Connect(30000); !ok {
 		gLog.Println(LvERROR, "P2PNetwork login error")
 		return nil
 	}
-	gLog.Println(LvINFO, "waiting for connection...")
-	return network
+	// gLog.Println(LvINFO, "waiting for connection...")
+	return GNetwork
+}
+
+func RunCmd(cmd string) {
+	rand.Seed(time.Now().UnixNano())
+	baseDir := filepath.Dir(os.Args[0])
+	os.Chdir(baseDir) // for system service
+	gLog = NewLogger(baseDir, ProductName, LvINFO, 1024*1024, LogFile|LogConsole)
+
+	parseParams("", cmd)
+	setFirewall()
+	err := setRLimit()
+	if err != nil {
+		gLog.Println(LvINFO, "setRLimit error:", err)
+	}
+	GNetwork = P2PNetworkInstance()
+	if ok := GNetwork.Connect(30000); !ok {
+		gLog.Println(LvERROR, "P2PNetwork login error")
+		return
+	}
+	forever := make(chan bool)
+	<-forever
 }
 
 func GetToken(baseDir string) string {
@@ -91,3 +115,7 @@ func GetToken(baseDir string) string {
 	gConf.load()
 	return fmt.Sprintf("%d", gConf.Network.Token)
 }
+
+func Stop() {
+	os.Exit(0)
+}

core/optun.go

@@ -0,0 +1,20 @@
+package openp2p
+
+import (
+	"github.com/openp2p-cn/wireguard-go/tun"
+)
+
+var AndroidSDWANConfig chan []byte
+
+type optun struct {
+	tunName string
+	dev     tun.Device
+}
+
+func (t *optun) Stop() error {
+	t.dev.Close()
+	return nil
+}
+func init() {
+	AndroidSDWANConfig = make(chan []byte, 1)
+}

core/optun_android.go

@@ -0,0 +1,85 @@
+// optun_android.go
+//go:build android
+// +build android
+
+package openp2p
+
+import (
+	"net"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+var AndroidReadTun chan []byte // TODO: multi channel
+var AndroidWriteTun chan []byte
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	bufs[0] = <-AndroidReadTun
+	sizes[0] = len(bufs[0])
+	return 1, nil
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	AndroidWriteTun <- bufs[0]
+	return len(bufs[0]), nil
+}
+
+func AndroidRead(data []byte, len int) {
+	head := PacketHeader{}
+	parseHeader(data, &head)
+	gLog.Printf(LvDev, "AndroidRead tun dst ip=%s,len=%d", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len)
+	buf := make([]byte, len)
+	copy(buf, data)
+	AndroidReadTun <- buf
+}
+
+func AndroidWrite(buf []byte) int {
+	p := <-AndroidWriteTun
+	copy(buf, p)
+	return len(p)
+}
+
+func GetAndroidSDWANConfig(buf []byte) int {
+	p := <-AndroidSDWANConfig
+	copy(buf, p)
+	gLog.Printf(LvINFO, "AndroidSDWANConfig=%s", p)
+	return len(p)
+}
+
+func GetAndroidNodeName() string {
+	gLog.Printf(LvINFO, "GetAndroidNodeName=%s", gConf.Network.Node)
+	return gConf.Network.Node
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	// TODO:
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	// TODO:
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	// TODO:
+	return nil
+}
+
+func delRoutesByGateway(gateway string) error {
+	// TODO:
+	return nil
+}
+
+func init() {
+	AndroidReadTun = make(chan []byte, 1000)
+	AndroidWriteTun = make(chan []byte, 1000)
+}

core/optun_darwin.go

@@ -0,0 +1,88 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"os/exec"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+)
+
+const (
+	tunIfaceName = "utun"
+	PIHeaderSize = 4 // utun has no IFF_NO_PI
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+	if err != nil {
+		return err
+	}
+	t.tunName, _ = t.dev.Name()
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	li, _, err := net.ParseCIDR(localAddr)
+	if err != nil {
+		return fmt.Errorf("parse local addr fail:%s", err)
+	}
+	ri, _, err := net.ParseCIDR(remoteAddr)
+	if err != nil {
+		return fmt.Errorf("parse remote addr fail:%s", err)
+	}
+	err = exec.Command("ifconfig", ifname, "inet", li.String(), ri.String(), "up").Run()
+	return err
+}
+
+func addRoute(dst, gw, ifname string) error {
+	err := exec.Command("route", "add", dst, gw).Run()
+	return err
+}
+
+func delRoute(dst, gw string) error {
+	err := exec.Command("route", "delete", dst, "-gateway", gw).Run()
+	return err
+}
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("netstat", "-rn")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 2 {
+			cmd := exec.Command("route", "delete", fields[0], gateway)
+			err := cmd.Run()
+			if err != nil {
+				gLog.Printf(LvERROR, "Delete route %s error:%s", fields[0], err)
+				continue
+			}
+			gLog.Printf(LvINFO, "Delete route ok: %s %s\n", fields[0], gateway)
+		}
+	}
+	return nil
+}
+func addTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}
+func delTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}

core/optun_linux.go

@@ -0,0 +1,134 @@
+//go:build !android
+// +build !android
+
+// optun_linux.go
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"os/exec"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+	"github.com/vishvananda/netlink"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+var previousIP = ""
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	ifce, err := netlink.LinkByName(ifname)
+	if err != nil {
+		return err
+	}
+	netlink.LinkSetMTU(ifce, 1375)
+	netlink.LinkSetTxQLen(ifce, 100)
+	netlink.LinkSetUp(ifce)
+
+	ln, err := netlink.ParseIPNet(localAddr)
+	if err != nil {
+		return err
+	}
+	ln.Mask = net.CIDRMask(32, 32)
+	rn, err := netlink.ParseIPNet(remoteAddr)
+	if err != nil {
+		return err
+	}
+	rn.Mask = net.CIDRMask(32, 32)
+
+	addr := &netlink.Addr{
+		IPNet: ln,
+		Peer:  rn,
+	}
+	if previousIP != "" {
+		lnDel, err := netlink.ParseIPNet(previousIP)
+		if err != nil {
+			return err
+		}
+		lnDel.Mask = net.CIDRMask(32, 32)
+
+		addrDel := &netlink.Addr{
+			IPNet: lnDel,
+			Peer:  rn,
+		}
+		netlink.AddrDel(ifce, addrDel)
+	}
+	previousIP = localAddr
+	return netlink.AddrAdd(ifce, addr)
+}
+
+func addRoute(dst, gw, ifname string) error {
+	_, networkid, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	ipGW := net.ParseIP(gw)
+	if ipGW == nil {
+		return fmt.Errorf("parse gateway %s failed", gw)
+	}
+	route := &netlink.Route{
+		Dst: networkid,
+		Gw:  ipGW,
+	}
+	return netlink.RouteAdd(route)
+}
+
+func delRoute(dst, gw string) error {
+	_, networkid, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	route := &netlink.Route{
+		Dst: networkid,
+	}
+	return netlink.RouteDel(route)
+}
+
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("route", "-n")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 8 && fields[1] == "0.0.0.0" && fields[7] == gateway {
+			delCmd := exec.Command("route", "del", "-net", fields[0], "gw", gateway)
+			err := delCmd.Run()
+			if err != nil {
+				gLog.Printf(LvERROR, "Delete route %s error:%s", fields[0], err)
+				continue
+			}
+			gLog.Printf(LvINFO, "Delete route ok: %s %s %s\n", fields[0], fields[1], gateway)
+		}
+	}
+	return nil
+}

core/optun_other.go

@@ -0,0 +1,42 @@
+//go:build !linux && !windows && !darwin
+// +build !linux,!windows,!darwin
+
+package openp2p
+
+import "github.com/openp2p-cn/wireguard-go/tun"
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+
+	if err != nil {
+		return err
+	}
+	err = setTunAddr(t.tunName, localAddr, detail.Gateway, t.dev)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	return nil
+}
+func addTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}
+
+func delTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}

core/optun_windows.go

@@ -0,0 +1,143 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"net/netip"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+	"golang.org/x/sys/windows"
+	"golang.zx2c4.com/wireguard/windows/tunnel/winipcfg"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	// check wintun.dll
+	tmpFile := filepath.Dir(os.Args[0]) + "/wintun.dll"
+	fs, err := os.Stat(tmpFile)
+	if err != nil || fs.Size() == 0 {
+		url := fmt.Sprintf("https://openp2p.cn/download/v1/latest/wintun/%s/wintun.dll", runtime.GOARCH)
+		err = downloadFile(url, "", tmpFile)
+		if err != nil {
+			os.Remove(tmpFile)
+			return err
+		}
+	}
+
+	t.tunName = tunIfaceName
+
+	uuid := &windows.GUID{
+		Data1: 0xf411e821,
+		Data2: 0xb310,
+		Data3: 0x4567,
+		Data4: [8]byte{0x80, 0x42, 0x83, 0x7e, 0xf4, 0x56, 0xce, 0x13},
+	}
+	t.dev, err = tun.CreateTUNWithRequestedGUID(t.tunName, uuid, 1420)
+	if err != nil { // retry
+		t.dev, err = tun.CreateTUNWithRequestedGUID(t.tunName, uuid, 1420)
+	}
+
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	nativeTunDevice := wintun.(*tun.NativeTun)
+	link := winipcfg.LUID(nativeTunDevice.LUID())
+	ip, err := netip.ParsePrefix(localAddr)
+	if err != nil {
+		gLog.Printf(LvERROR, "ParsePrefix error:%s, luid:%d,localAddr:%s", err, nativeTunDevice.LUID(), localAddr)
+		return err
+	}
+	err = link.SetIPAddresses([]netip.Prefix{ip})
+	if err != nil {
+		gLog.Printf(LvERROR, "SetIPAddresses error:%s, netip.Prefix:%+v", err, []netip.Prefix{ip})
+		return err
+	}
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	_, dstNet, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	i, err := net.InterfaceByName(ifname)
+	if err != nil {
+		return err
+	}
+	params := make([]string, 0)
+	params = append(params, "add")
+	params = append(params, dstNet.IP.String())
+	params = append(params, "mask")
+	params = append(params, net.IP(dstNet.Mask).String())
+	params = append(params, gw)
+	params = append(params, "if")
+	params = append(params, strconv.Itoa(i.Index))
+	// gLogger.Println(LevelINFO, "windows add route params:", params)
+	execCommand("route", true, params...)
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	_, dstNet, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	params := make([]string, 0)
+	params = append(params, "delete")
+	params = append(params, dstNet.IP.String())
+	params = append(params, "mask")
+	params = append(params, net.IP(dstNet.Mask).String())
+	params = append(params, gw)
+	// gLogger.Println(LevelINFO, "windows delete route params:", params)
+	execCommand("route", true, params...)
+	return nil
+}
+
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("route", "print", "-4")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 5 {
+			cmd := exec.Command("route", "delete", fields[0], "mask", fields[1], gateway)
+			err := cmd.Run()
+			if err != nil {
+				gLog.Printf(LvERROR, "Delete route %s error:%s", fields[0], err)
+				continue
+			}
+			gLog.Printf(LvINFO, "Delete route ok: %s %s %s\n", fields[0], fields[1], gateway)
+		}
+	}
+	return nil
+}

core/overlay.go

@@ -23,15 +23,16 @@ func (e *DeadlineExceededError) Temporary() bool { return true }
 
 // implement io.Writer
 type overlayConn struct {
-	tunnel      *P2PTunnel
+	tunnel      *P2PTunnel // TODO: del
+	app         *p2pApp
 	connTCP     net.Conn
 	id          uint64
 	rtid        uint64
 	running     bool
 	isClient    bool
-	appID       uint64
-	appKey      uint64
-	appKeyBytes []byte
+	appID       uint64 // TODO: del
+	appKey      uint64 // TODO: del
+	appKeyBytes []byte // TODO: del
 	// for udp
 	connUDP       *net.UDPConn
 	remoteAddr    net.Addr
@@ -65,15 +66,16 @@ func (oConn *overlayConn) run() {
 			payload, _ = encryptBytes(oConn.appKeyBytes, encryptData, readBuff[:dataLen], dataLen)
 		}
 		writeBytes := append(tunnelHead.Bytes(), payload...)
+		// TODO: app.write
 		if oConn.rtid == 0 {
 			oConn.tunnel.conn.WriteBytes(MsgP2P, MsgOverlayData, writeBytes)
-			gLog.Printf(LvDEBUG, "write overlay data to %d:%d bodylen=%d", oConn.rtid, oConn.id, len(writeBytes))
+			gLog.Printf(LvDev, "write overlay data to tid:%d,oid:%d bodylen=%d", oConn.tunnel.id, oConn.id, len(writeBytes))
 		} else {
 			// write raley data
 			all := append(relayHead.Bytes(), encodeHeader(MsgP2P, MsgOverlayData, uint32(len(writeBytes)))...)
 			all = append(all, writeBytes...)
 			oConn.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, all)
-			gLog.Printf(LvDEBUG, "write relay data to %d:%d bodylen=%d", oConn.rtid, oConn.id, len(writeBytes))
+			gLog.Printf(LvDev, "write relay data to tid:%d,rtid:%d,oid:%d bodylen=%d", oConn.tunnel.id, oConn.rtid, oConn.id, len(writeBytes))
 		}
 	}
 	if oConn.connTCP != nil {
@@ -85,14 +87,7 @@ func (oConn *overlayConn) run() {
 	oConn.tunnel.overlayConns.Delete(oConn.id)
 	// notify peer disconnect
 	req := OverlayDisconnectReq{ID: oConn.id}
-	if oConn.rtid == 0 {
-		oConn.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayDisconnectReq, &req)
-	} else {
-		// write relay data
-		msg, _ := newMessage(MsgP2P, MsgOverlayDisconnectReq, &req)
-		msgWithHead := append(relayHead.Bytes(), msg...)
-		oConn.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
-	}
+	oConn.tunnel.WriteMessage(oConn.rtid, MsgP2P, MsgOverlayDisconnectReq, &req)
 }
 
 func (oConn *overlayConn) Read(reuseBuff []byte) (buff []byte, dataLen int, err error) {
@@ -163,11 +158,11 @@ func (oConn *overlayConn) Close() (err error) {
 	oConn.running = false
 	if oConn.connTCP != nil {
 		oConn.connTCP.Close()
-		oConn.connTCP = nil
+		// oConn.connTCP = nil
 	}
 	if oConn.connUDP != nil {
 		oConn.connUDP.Close()
-		oConn.connUDP = nil
+		// oConn.connUDP = nil
 	}
 	return nil
 }

core/p2papp.go

@@ -16,47 +16,383 @@ type p2pApp struct {
 	config       AppConfig
 	listener     net.Listener
 	listenerUDP  *net.UDPConn
-	tunnel      *P2PTunnel
-	iptree      *IPTree
+	directTunnel *P2PTunnel
+	relayTunnel  *P2PTunnel
+	tunnelMtx    sync.Mutex
+	iptree       *IPTree // for whitelist
 	rtid         uint64  // relay tunnelID
 	relayNode    string
-	relayMode   string
-	hbTime      time.Time
+	relayMode    string // public/private
+	hbTimeRelay  time.Time
 	hbMtx        sync.Mutex
 	running      bool
 	id           uint64
-	key         uint64
+	key          uint64 // aes
 	wg           sync.WaitGroup
+	relayHead    *bytes.Buffer
+	once         sync.Once
+	// for relayTunnel
+	retryRelayNum      int
+	retryRelayTime     time.Time
+	nextRetryRelayTime time.Time
+	errMsg             string
+	connectTime        time.Time
+}
+
+func (app *p2pApp) Tunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	if app.directTunnel != nil {
+		return app.directTunnel
+	}
+	return app.relayTunnel
+}
+
+func (app *p2pApp) DirectTunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	return app.directTunnel
+}
+
+func (app *p2pApp) setDirectTunnel(t *P2PTunnel) {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	app.directTunnel = t
+}
+
+func (app *p2pApp) RelayTunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	return app.relayTunnel
+}
+
+func (app *p2pApp) setRelayTunnel(t *P2PTunnel) {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	app.relayTunnel = t
+}
+
+func (app *p2pApp) isDirect() bool {
+	return app.directTunnel != nil
+}
+
+func (app *p2pApp) RelayTunnelID() uint64 {
+	if app.isDirect() {
+		return 0
+	}
+	return app.rtid
+}
+
+func (app *p2pApp) ConnectTime() time.Time {
+	if app.isDirect() {
+		return app.config.connectTime
+	}
+	return app.connectTime
+}
+
+func (app *p2pApp) RetryTime() time.Time {
+	if app.isDirect() {
+		return app.config.retryTime
+	}
+	return app.retryRelayTime
+}
+
+func (app *p2pApp) checkP2PTunnel() error {
+	for app.running {
+		app.checkDirectTunnel()
+		app.checkRelayTunnel()
+		time.Sleep(time.Second * 3)
+	}
+	return nil
+}
+
+func (app *p2pApp) directRetryLimit() int {
+	if app.config.peerIP == gConf.Network.publicIP && compareVersion(app.config.peerVersion, SupportIntranetVersion) >= 0 {
+		return retryLimit
+	}
+	if IsIPv6(app.config.peerIPv6) && IsIPv6(gConf.IPv6()) {
+		return retryLimit
+	}
+	if app.config.hasIPv4 == 1 || gConf.Network.hasIPv4 == 1 || app.config.hasUPNPorNATPMP == 1 || gConf.Network.hasUPNPorNATPMP == 1 {
+		return retryLimit
+	}
+	if gConf.Network.natType == NATCone && app.config.peerNatType == NATCone {
+		return retryLimit
+	}
+	if app.config.peerNatType == NATSymmetric && gConf.Network.natType == NATSymmetric {
+		return 0
+	}
+	return retryLimit / 10 // c2s or s2c
+}
+func (app *p2pApp) checkDirectTunnel() error {
+	if app.config.ForceRelay == 1 && app.config.RelayNode != app.config.PeerNode {
+		return nil
+	}
+	if app.DirectTunnel() != nil && app.DirectTunnel().isActive() {
+		return nil
+	}
+	if app.config.nextRetryTime.After(time.Now()) || app.config.Enabled == 0 || app.config.retryNum >= app.directRetryLimit() {
+		return nil
+	}
+	if time.Now().Add(-time.Minute * 15).After(app.config.retryTime) { // run normally 15min, reset retrynum
+		app.config.retryNum = 1
+	}
+	if app.config.retryNum > 0 { // first time not show reconnect log
+		gLog.Printf(LvINFO, "detect app %s appid:%d disconnect, reconnecting the %d times...", app.config.LogPeerNode(), app.id, app.config.retryNum)
+	}
+	app.config.retryNum++
+	app.config.retryTime = time.Now()
+	app.config.nextRetryTime = time.Now().Add(retryInterval)
+	app.config.connectTime = time.Now()
+	err := app.buildDirectTunnel()
+	if err != nil {
+		app.config.errMsg = err.Error()
+		if err == ErrPeerOffline && app.config.retryNum > 2 { // stop retry, waiting for online
+			app.config.retryNum = retryLimit
+			gLog.Printf(LvINFO, " %s offline, it will auto reconnect when peer node online", app.config.LogPeerNode())
+		}
+		if err == ErrBuildTunnelBusy {
+			app.config.retryNum--
+		}
+	}
+	if app.Tunnel() != nil {
+		app.once.Do(func() {
+			go app.listen()
+			// memapp also need
+			go app.relayHeartbeatLoop()
+		})
+	}
+	return nil
+}
+func (app *p2pApp) buildDirectTunnel() error {
+	relayNode := ""
+	peerNatType := NATUnknown
+	peerIP := ""
+	errMsg := ""
+	var t *P2PTunnel
+	var err error
+	pn := GNetwork
+	initErr := pn.requestPeerInfo(&app.config)
+	if initErr != nil {
+		gLog.Printf(LvERROR, "%s requestPeerInfo error:%s", app.config.LogPeerNode(), initErr)
+		return initErr
+	}
+	t, err = pn.addDirectTunnel(app.config, 0)
+	if t != nil {
+		peerNatType = t.config.peerNatType
+		peerIP = t.config.peerIP
+	}
+	if err != nil {
+		errMsg = err.Error()
+	}
+	req := ReportConnect{
+		Error:          errMsg,
+		Protocol:       app.config.Protocol,
+		SrcPort:        app.config.SrcPort,
+		NatType:        gConf.Network.natType,
+		PeerNode:       app.config.PeerNode,
+		DstPort:        app.config.DstPort,
+		DstHost:        app.config.DstHost,
+		PeerNatType:    peerNatType,
+		PeerIP:         peerIP,
+		ShareBandwidth: gConf.Network.ShareBandwidth,
+		RelayNode:      relayNode,
+		Version:        OpenP2PVersion,
+	}
+	pn.write(MsgReport, MsgReportConnect, &req)
+	if err != nil {
+		return err
+	}
+	// if rtid != 0 || t.conn.Protocol() == "tcp" {
+	// sync appkey
+	if t == nil {
+		return err
+	}
+	syncKeyReq := APPKeySync{
+		AppID:  app.id,
+		AppKey: app.key,
+	}
+	gLog.Printf(LvDEBUG, "sync appkey direct to %s", app.config.LogPeerNode())
+	pn.push(app.config.PeerNode, MsgPushAPPKey, &syncKeyReq)
+	app.setDirectTunnel(t)
+
+	// if memapp notify peer addmemapp
+	if app.config.SrcPort == 0 {
+		req := ServerSideSaveMemApp{From: gConf.Network.Node, Node: gConf.Network.Node, TunnelID: t.id, RelayTunnelID: 0, AppID: app.id}
+		pn.push(app.config.PeerNode, MsgPushServerSideSaveMemApp, &req)
+		gLog.Printf(LvDEBUG, "push %s ServerSideSaveMemApp: %s", app.config.LogPeerNode(), prettyJson(req))
+	}
+	gLog.Printf(LvDEBUG, "%s use tunnel %d", app.config.AppName, t.id)
+	return nil
+}
+
+func (app *p2pApp) checkRelayTunnel() error {
+	// if app.config.ForceRelay == 1 && (gConf.sdwan.CentralNode == app.config.PeerNode && compareVersion(app.config.peerVersion, SupportDualTunnelVersion) < 0) {
+	if app.config.SrcPort == 0 && (gConf.sdwan.CentralNode == app.config.PeerNode || gConf.sdwan.CentralNode == gConf.Network.Node) { // memapp central node not build relay tunnel
+		return nil
+	}
+	app.hbMtx.Lock()
+	if app.RelayTunnel() != nil && time.Now().Before(app.hbTimeRelay.Add(TunnelHeartbeatTime*2)) { // must check app.hbtime instead of relayTunnel
+		app.hbMtx.Unlock()
+		return nil
+	}
+	app.hbMtx.Unlock()
+	if app.nextRetryRelayTime.After(time.Now()) || app.config.Enabled == 0 || app.retryRelayNum >= retryLimit {
+		return nil
+	}
+	if time.Now().Add(-time.Minute * 15).After(app.retryRelayTime) { // run normally 15min, reset retrynum
+		app.retryRelayNum = 1
+	}
+	if app.retryRelayNum > 0 { // first time not show reconnect log
+		gLog.Printf(LvINFO, "detect app %s appid:%d relay disconnect, reconnecting the %d times...", app.config.LogPeerNode(), app.id, app.retryRelayNum)
+	}
+	app.setRelayTunnel(nil) // reset relayTunnel
+	app.retryRelayNum++
+	app.retryRelayTime = time.Now()
+	app.nextRetryRelayTime = time.Now().Add(retryInterval)
+	app.connectTime = time.Now()
+	err := app.buildRelayTunnel()
+	if err != nil {
+		app.errMsg = err.Error()
+		if err == ErrPeerOffline && app.retryRelayNum > 2 { // stop retry, waiting for online
+			app.retryRelayNum = retryLimit
+			gLog.Printf(LvINFO, " %s offline, it will auto reconnect when peer node online", app.config.LogPeerNode())
+		}
+	}
+	if app.Tunnel() != nil {
+		app.once.Do(func() {
+			go app.listen()
+			// memapp also need
+			go app.relayHeartbeatLoop()
+		})
+	}
+	return nil
+}
+
+func (app *p2pApp) buildRelayTunnel() error {
+	var rtid uint64
+	relayNode := ""
+	relayMode := ""
+	peerNatType := NATUnknown
+	peerIP := ""
+	errMsg := ""
+	var t *P2PTunnel
+	var err error
+	pn := GNetwork
+	config := app.config
+	initErr := pn.requestPeerInfo(&config)
+	if initErr != nil {
+		gLog.Printf(LvERROR, "%s init error:%s", config.LogPeerNode(), initErr)
+		return initErr
+	}
+
+	t, rtid, relayMode, err = pn.addRelayTunnel(config)
+	if t != nil {
+		relayNode = t.config.PeerNode
+	}
+
+	if err != nil {
+		errMsg = err.Error()
+	}
+	req := ReportConnect{
+		Error:          errMsg,
+		Protocol:       config.Protocol,
+		SrcPort:        config.SrcPort,
+		NatType:        gConf.Network.natType,
+		PeerNode:       config.PeerNode,
+		DstPort:        config.DstPort,
+		DstHost:        config.DstHost,
+		PeerNatType:    peerNatType,
+		PeerIP:         peerIP,
+		ShareBandwidth: gConf.Network.ShareBandwidth,
+		RelayNode:      relayNode,
+		Version:        OpenP2PVersion,
+	}
+	pn.write(MsgReport, MsgReportConnect, &req)
+	if err != nil {
+		return err
+	}
+	// if rtid != 0 || t.conn.Protocol() == "tcp" {
+	// sync appkey
+	syncKeyReq := APPKeySync{
+		AppID:  app.id,
+		AppKey: app.key,
+	}
+	gLog.Printf(LvDEBUG, "sync appkey relay to %s", config.LogPeerNode())
+	pn.push(config.PeerNode, MsgPushAPPKey, &syncKeyReq)
+	app.setRelayTunnelID(rtid)
+	app.setRelayTunnel(t)
+	app.relayNode = relayNode
+	app.relayMode = relayMode
+	app.hbTimeRelay = time.Now()
+
+	// if memapp notify peer addmemapp
+	if config.SrcPort == 0 {
+		req := ServerSideSaveMemApp{From: gConf.Network.Node, Node: relayNode, TunnelID: rtid, RelayTunnelID: t.id, AppID: app.id, RelayMode: relayMode}
+		pn.push(config.PeerNode, MsgPushServerSideSaveMemApp, &req)
+		gLog.Printf(LvDEBUG, "push %s relay ServerSideSaveMemApp: %s", config.LogPeerNode(), prettyJson(req))
+	}
+	gLog.Printf(LvDEBUG, "%s use tunnel %d", app.config.AppName, t.id)
+	return nil
+}
+
+func (app *p2pApp) buildOfficialTunnel() error {
+	return nil
+}
+
+// cache relayHead, refresh when rtid change
+func (app *p2pApp) RelayHead() *bytes.Buffer {
+	if app.relayHead == nil {
+		app.relayHead = new(bytes.Buffer)
+		binary.Write(app.relayHead, binary.LittleEndian, app.rtid)
+	}
+	return app.relayHead
+}
+
+func (app *p2pApp) setRelayTunnelID(rtid uint64) {
+	app.rtid = rtid
+	app.relayHead = new(bytes.Buffer)
+	binary.Write(app.relayHead, binary.LittleEndian, app.rtid)
 }
 
 func (app *p2pApp) isActive() bool {
-	if app.tunnel == nil {
+	if app.Tunnel() == nil {
+		// gLog.Printf(LvDEBUG, "isActive app.tunnel==nil")
 		return false
 	}
-	if app.rtid == 0 { // direct mode app heartbeat equals to tunnel heartbeat
-		return app.tunnel.isActive()
+	if app.isDirect() { // direct mode app heartbeat equals to tunnel heartbeat
+		return app.Tunnel().isActive()
 	}
 	// relay mode calc app heartbeat
 	app.hbMtx.Lock()
 	defer app.hbMtx.Unlock()
-	return time.Now().Before(app.hbTime.Add(TunnelIdleTimeout))
+	res := time.Now().Before(app.hbTimeRelay.Add(TunnelHeartbeatTime * 2))
+	// if !res {
+	// 	gLog.Printf(LvDEBUG, "%d app isActive false. peer=%s", app.id, app.config.PeerNode)
+	// }
+	return res
 }
 
 func (app *p2pApp) updateHeartbeat() {
 	app.hbMtx.Lock()
 	defer app.hbMtx.Unlock()
-	app.hbTime = time.Now()
+	app.hbTimeRelay = time.Now()
 }
 
 func (app *p2pApp) listenTCP() error {
 	gLog.Printf(LvDEBUG, "tcp accept on port %d start", app.config.SrcPort)
 	defer gLog.Printf(LvDEBUG, "tcp accept on port %d end", app.config.SrcPort)
 	var err error
-	app.listener, err = net.Listen("tcp", fmt.Sprintf("0.0.0.0:%d", app.config.SrcPort)) // support tcp4 and tcp6
+	listenAddr := ""
+	if IsLocalhost(app.config.Whitelist) { // not expose port
+		listenAddr = "127.0.0.1"
+	}
+	app.listener, err = net.Listen("tcp", fmt.Sprintf("%s:%d", listenAddr, app.config.SrcPort))
 	if err != nil {
 		gLog.Printf(LvERROR, "listen error:%s", err)
 		return err
 	}
+	defer app.listener.Close()
 	for app.running {
 		conn, err := app.listener.Accept()
 		if err != nil {
@@ -65,25 +401,33 @@ func (app *p2pApp) listenTCP() error {
 			}
 			break
 		}
+		if app.Tunnel() == nil {
+			gLog.Printf(LvDEBUG, "srcPort=%d, app.Tunnel()==nil, not ready", app.config.SrcPort)
+			time.Sleep(time.Second)
+			continue
+		}
 		// check white list
 		if app.config.Whitelist != "" {
-			remoteIP := strings.Split(conn.RemoteAddr().String(), ":")[0]
-			if !app.iptree.Contains(remoteIP) {
+			remoteIP := conn.RemoteAddr().(*net.TCPAddr).IP.String()
+			if !app.iptree.Contains(remoteIP) && !IsLocalhost(remoteIP) {
 				conn.Close()
 				gLog.Printf(LvERROR, "%s not in whitelist, access denied", remoteIP)
 				continue
 			}
 		}
 		oConn := overlayConn{
-			tunnel:   app.tunnel,
+			tunnel:   app.Tunnel(),
+			app:      app,
 			connTCP:  conn,
 			id:       rand.Uint64(),
 			isClient: true,
-			rtid:     app.rtid,
 			appID:    app.id,
 			appKey:   app.key,
 			running:  true,
 		}
+		if !app.isDirect() {
+			oConn.rtid = app.rtid
+		}
 		// pre-calc key bytes for encrypt
 		if oConn.appKey != 0 {
 			encryptKey := make([]byte, AESKeySize)
@@ -91,26 +435,20 @@ func (app *p2pApp) listenTCP() error {
 			binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
 			oConn.appKeyBytes = encryptKey
 		}
-		app.tunnel.overlayConns.Store(oConn.id, &oConn)
+		app.Tunnel().overlayConns.Store(oConn.id, &oConn)
 		gLog.Printf(LvDEBUG, "Accept TCP overlayID:%d, %s", oConn.id, oConn.connTCP.RemoteAddr())
 		// tell peer connect
 		req := OverlayConnectReq{ID: oConn.id,
-			Token:    app.tunnel.pn.config.Token,
+			Token:    gConf.Network.Token,
 			DstIP:    app.config.DstHost,
 			DstPort:  app.config.DstPort,
 			Protocol: app.config.Protocol,
 			AppID:    app.id,
 		}
-		if app.rtid == 0 {
-			app.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayConnectReq, &req)
-		} else {
-			req.RelayTunnelID = app.tunnel.id
-			relayHead := new(bytes.Buffer)
-			binary.Write(relayHead, binary.LittleEndian, app.rtid)
-			msg, _ := newMessage(MsgP2P, MsgOverlayConnectReq, &req)
-			msgWithHead := append(relayHead.Bytes(), msg...)
-			app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+		if !app.isDirect() {
+			req.RelayTunnelID = app.Tunnel().id
 		}
+		app.Tunnel().WriteMessage(app.RelayTunnelID(), MsgP2P, MsgOverlayConnectReq, &req)
 		// TODO: wait OverlayConnectRsp instead of sleep
 		time.Sleep(time.Second) // waiting remote node connection ok
 		go oConn.run()
@@ -127,6 +465,7 @@ func (app *p2pApp) listenUDP() error {
 		gLog.Printf(LvERROR, "listen error:%s", err)
 		return err
 	}
+	defer app.listenerUDP.Close()
 	buffer := make([]byte, 64*1024+PaddingSize)
 	udpID := make([]byte, 8)
 	for {
@@ -140,6 +479,11 @@ func (app *p2pApp) listenUDP() error {
 				break
 			}
 		} else {
+			if app.Tunnel() == nil {
+				gLog.Printf(LvDEBUG, "srcPort=%d, app.Tunnel()==nil, not ready", app.config.SrcPort)
+				time.Sleep(time.Second)
+				continue
+			}
 			dupData := bytes.Buffer{} // should uses memory pool
 			dupData.Write(buffer[:len+PaddingSize])
 			// load from app.tunnel.overlayConns by remoteAddr error, new udp connection
@@ -153,20 +497,22 @@ func (app *p2pApp) listenUDP() error {
 			udpID[4] = byte(port)
 			udpID[5] = byte(port >> 8)
 			id := binary.LittleEndian.Uint64(udpID) // convert remoteIP:port to uint64
-			s, ok := app.tunnel.overlayConns.Load(id)
+			s, ok := app.Tunnel().overlayConns.Load(id)
 			if !ok {
 				oConn := overlayConn{
-					tunnel:     app.tunnel,
+					tunnel:     app.Tunnel(),
 					connUDP:    app.listenerUDP,
 					remoteAddr: remoteAddr,
 					udpData:    make(chan []byte, 1000),
 					id:         id,
 					isClient:   true,
-					rtid:       app.rtid,
 					appID:      app.id,
 					appKey:     app.key,
 					running:    true,
 				}
+				if !app.isDirect() {
+					oConn.rtid = app.rtid
+				}
 				// calc key bytes for encrypt
 				if oConn.appKey != 0 {
 					encryptKey := make([]byte, AESKeySize)
@@ -174,26 +520,20 @@ func (app *p2pApp) listenUDP() error {
 					binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
 					oConn.appKeyBytes = encryptKey
 				}
-				app.tunnel.overlayConns.Store(oConn.id, &oConn)
+				app.Tunnel().overlayConns.Store(oConn.id, &oConn)
 				gLog.Printf(LvDEBUG, "Accept UDP overlayID:%d", oConn.id)
 				// tell peer connect
 				req := OverlayConnectReq{ID: oConn.id,
-					Token:    app.tunnel.pn.config.Token,
+					Token:    gConf.Network.Token,
 					DstIP:    app.config.DstHost,
 					DstPort:  app.config.DstPort,
 					Protocol: app.config.Protocol,
 					AppID:    app.id,
 				}
-				if app.rtid == 0 {
-					app.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayConnectReq, &req)
-				} else {
-					req.RelayTunnelID = app.tunnel.id
-					relayHead := new(bytes.Buffer)
-					binary.Write(relayHead, binary.LittleEndian, app.rtid)
-					msg, _ := newMessage(MsgP2P, MsgOverlayConnectReq, &req)
-					msgWithHead := append(relayHead.Bytes(), msg...)
-					app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+				if !app.isDirect() {
+					req.RelayTunnelID = app.Tunnel().id
 				}
+				app.Tunnel().WriteMessage(app.RelayTunnelID(), MsgP2P, MsgOverlayConnectReq, &req)
 				// TODO: wait OverlayConnectRsp instead of sleep
 				time.Sleep(time.Second) // waiting remote node connection ok
 				go oConn.run()
@@ -212,15 +552,14 @@ func (app *p2pApp) listenUDP() error {
 }
 
 func (app *p2pApp) listen() error {
+	if app.config.SrcPort == 0 {
+		return nil
+	}
 	gLog.Printf(LvINFO, "LISTEN ON PORT %s:%d START", app.config.Protocol, app.config.SrcPort)
 	defer gLog.Printf(LvINFO, "LISTEN ON PORT %s:%d END", app.config.Protocol, app.config.SrcPort)
 	app.wg.Add(1)
 	defer app.wg.Done()
-	app.running = true
-	if app.rtid != 0 {
-		go app.relayHeartbeatLoop()
-	}
-	for app.tunnel.isRuning() {
+	for app.running {
 		if app.config.Protocol == "udp" {
 			app.listenUDP()
 		} else {
@@ -242,8 +581,11 @@ func (app *p2pApp) close() {
 	if app.listenerUDP != nil {
 		app.listenerUDP.Close()
 	}
-	if app.tunnel != nil {
-		app.tunnel.closeOverlayConns(app.id)
+	if app.DirectTunnel() != nil {
+		app.DirectTunnel().closeOverlayConns(app.id)
+	}
+	if app.RelayTunnel() != nil {
+		app.RelayTunnel().closeOverlayConns(app.id)
 	}
 	app.wg.Wait()
 }
@@ -252,16 +594,23 @@ func (app *p2pApp) close() {
 func (app *p2pApp) relayHeartbeatLoop() {
 	app.wg.Add(1)
 	defer app.wg.Done()
-	gLog.Printf(LvDEBUG, "relayHeartbeat to %d start", app.rtid)
-	defer gLog.Printf(LvDEBUG, "relayHeartbeat to %d end", app.rtid)
-	relayHead := new(bytes.Buffer)
-	binary.Write(relayHead, binary.LittleEndian, app.rtid)
-	req := RelayHeartbeat{RelayTunnelID: app.tunnel.id,
+	gLog.Printf(LvDEBUG, "%s appid:%d relayHeartbeat to rtid:%d start", app.config.LogPeerNode(), app.id, app.rtid)
+	defer gLog.Printf(LvDEBUG, "%s appid:%d relayHeartbeat to rtid%d end", app.config.LogPeerNode(), app.id, app.rtid)
+
+	for app.running {
+		if app.RelayTunnel() == nil || !app.RelayTunnel().isRuning() {
+			time.Sleep(TunnelHeartbeatTime)
+			continue
+		}
+		req := RelayHeartbeat{From: gConf.Network.Node, RelayTunnelID: app.RelayTunnel().id,
 			AppID: app.id}
-	msg, _ := newMessage(MsgP2P, MsgRelayHeartbeat, &req)
-	msgWithHead := append(relayHead.Bytes(), msg...)
-	for app.tunnel.isRuning() && app.running {
-		app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+		err := app.RelayTunnel().WriteMessage(app.rtid, MsgP2P, MsgRelayHeartbeat, &req)
+		if err != nil {
+			gLog.Printf(LvERROR, "%s appid:%d rtid:%d write relay tunnel heartbeat error %s", app.config.LogPeerNode(), app.id, app.rtid, err)
+			return
+		}
+		// TODO: debug relay heartbeat
+		gLog.Printf(LvDEBUG, "%s appid:%d rtid:%d write relay tunnel heartbeat ok", app.config.LogPeerNode(), app.id, app.rtid)
 		time.Sleep(TunnelHeartbeatTime)
 	}
 }

core/p2ptunnel.go

@@ -10,20 +10,23 @@ import (
 	"net"
 	"reflect"
 	"sync"
+	"sync/atomic"
 	"time"
 )
 
+const WriteDataChanSize int = 3000
+
+var buildTunnelMtx sync.Mutex
+
 type P2PTunnel struct {
-	pn            *P2PNetwork
 	conn           underlay
 	hbTime         time.Time
 	hbMtx          sync.Mutex
-	hbTimeRelay   time.Time
 	config         AppConfig
-	la            *net.UDPAddr // local hole address
-	ra            *net.UDPAddr // remote hole address
+	localHoleAddr  *net.UDPAddr // local hole address
+	remoteHoleAddr *net.UDPAddr // remote hole address
 	overlayConns   sync.Map     // both TCP and UDP
-	id            uint64
+	id             uint64       // client side alloc rand.uint64 = server side
 	running        bool
 	runMtx         sync.Mutex
 	tunnelServer   bool // different from underlayServer
@@ -31,59 +34,58 @@ type P2PTunnel struct {
 	coneNatPort    int
 	linkModeWeb    string // use config.linkmode
 	punchTs        uint64
+	writeData      chan []byte
+	writeDataSmall chan []byte
 }
 
 func (t *P2PTunnel) initPort() {
 	t.running = true
-	t.hbMtx.Lock()
-	t.hbTime = time.Now()
-	t.hbMtx.Unlock()
-	t.hbTimeRelay = time.Now().Add(time.Second * 600) // TODO: test fake time
 	localPort := int(rand.Uint32()%15000 + 50000) // if the process has bug, will add many upnp port. use specify p2p port by param
-	if t.config.linkMode == LinkModeTCP6 || t.config.linkMode == LinkModeTCP4 {
-		t.coneLocalPort = t.pn.config.TCPPort
-		t.coneNatPort = t.pn.config.TCPPort // symmetric doesn't need coneNatPort
+	if t.config.linkMode == LinkModeTCP6 || t.config.linkMode == LinkModeTCP4 || t.config.linkMode == LinkModeIntranet {
+		t.coneLocalPort = gConf.Network.TCPPort
+		t.coneNatPort = gConf.Network.TCPPort // symmetric doesn't need coneNatPort
 	}
 	if t.config.linkMode == LinkModeUDPPunch {
 		// prepare one random cone hole manually
-		_, natPort, _ := natTest(t.pn.config.ServerHost, t.pn.config.UDPPort1, localPort)
+		_, natPort, _ := natTest(gConf.Network.ServerHost, gConf.Network.UDPPort1, localPort)
 		t.coneLocalPort = localPort
 		t.coneNatPort = natPort
 	}
 	if t.config.linkMode == LinkModeTCPPunch {
 		// prepare one random cone hole by system automatically
-		_, natPort, localPort2 := natTCP(t.pn.config.ServerHost, IfconfigPort1)
+		_, natPort, localPort2 := natTCP(gConf.Network.ServerHost, IfconfigPort1)
 		t.coneLocalPort = localPort2
 		t.coneNatPort = natPort
 	}
-	t.la = &net.UDPAddr{IP: net.ParseIP(t.pn.config.localIP), Port: t.coneLocalPort}
+	t.localHoleAddr = &net.UDPAddr{IP: net.ParseIP(gConf.Network.localIP), Port: t.coneLocalPort}
 	gLog.Printf(LvDEBUG, "prepare punching port %d:%d", t.coneLocalPort, t.coneNatPort)
 }
 
 func (t *P2PTunnel) connect() error {
-	gLog.Printf(LvDEBUG, "start p2pTunnel to %s ", t.config.PeerNode)
+	gLog.Printf(LvDEBUG, "start p2pTunnel to %s ", t.config.LogPeerNode())
 	t.tunnelServer = false
 	appKey := uint64(0)
 	req := PushConnectReq{
 		Token:            t.config.peerToken,
-		From:             t.pn.config.Node,
-		FromIP:           t.pn.config.publicIP,
+		From:             gConf.Network.Node,
+		FromIP:           gConf.Network.publicIP,
 		ConeNatPort:      t.coneNatPort,
-		NatType:          t.pn.config.natType,
-		HasIPv4:          t.pn.config.hasIPv4,
+		NatType:          gConf.Network.natType,
+		HasIPv4:          gConf.Network.hasIPv4,
 		IPv6:             gConf.IPv6(),
-		HasUPNPorNATPMP:  t.pn.config.hasUPNPorNATPMP,
+		HasUPNPorNATPMP:  gConf.Network.hasUPNPorNATPMP,
 		ID:               t.id,
 		AppKey:           appKey,
 		Version:          OpenP2PVersion,
 		LinkMode:         t.config.linkMode,
 		IsUnderlayServer: t.config.isUnderlayServer ^ 1, // peer
+		UnderlayProtocol: t.config.UnderlayProtocol,
 	}
 	if req.Token == 0 { // no relay token
-		req.Token = t.pn.config.Token
+		req.Token = gConf.Network.Token
 	}
-	t.pn.push(t.config.PeerNode, MsgPushConnectReq, req)
-	head, body := t.pn.read(t.config.PeerNode, MsgPush, MsgPushConnectRsp, HandshakeTimeout*3)
+	GNetwork.push(t.config.PeerNode, MsgPushConnectReq, req)
+	head, body := GNetwork.read(t.config.PeerNode, MsgPush, MsgPushConnectRsp, UnderlayConnectTimeout*3)
 	if head == nil {
 		return errors.New("connect error")
 	}
@@ -107,7 +109,6 @@ func (t *P2PTunnel) connect() error {
 	err := t.start()
 	if err != nil {
 		gLog.Println(LvERROR, "handshake error:", err)
-		err = ErrorHandshake
 	}
 	return err
 }
@@ -130,7 +131,11 @@ func (t *P2PTunnel) isActive() bool {
 	}
 	t.hbMtx.Lock()
 	defer t.hbMtx.Unlock()
-	return time.Now().Before(t.hbTime.Add(TunnelHeartbeatTime * 2))
+	res := time.Now().Before(t.hbTime.Add(TunnelHeartbeatTime * 2))
+	if !res {
+		gLog.Printf(LvDEBUG, "%d tunnel isActive false", t.id)
+	}
+	return res
 }
 
 func (t *P2PTunnel) checkActive() bool {
@@ -155,8 +160,16 @@ func (t *P2PTunnel) checkActive() bool {
 
 // call when user delete tunnel
 func (t *P2PTunnel) close() {
+	GNetwork.NotifyTunnelClose(t)
+	if !t.running {
+		return
+	}
 	t.setRun(false)
-	t.pn.allTunnels.Delete(t.id)
+	if t.conn != nil {
+		t.conn.Close()
+	}
+	GNetwork.allTunnels.Delete(t.id)
+	gLog.Printf(LvINFO, "%d p2ptunnel close %s ", t.id, t.config.LogPeerNode())
 }
 
 func (t *P2PTunnel) start() error {
@@ -176,29 +189,31 @@ func (t *P2PTunnel) start() error {
 func (t *P2PTunnel) handshake() error {
 	if t.config.peerConeNatPort > 0 { // only peer is cone should prepare t.ra
 		var err error
-		t.ra, err = net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, t.config.peerConeNatPort))
+		t.remoteHoleAddr, err = net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, t.config.peerConeNatPort))
 		if err != nil {
 			return err
 		}
 	}
-	if compareVersion(t.config.peerVersion, SyncServerTimeVersion) == LESS {
+	if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
 		gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
 	} else {
-		ts := time.Duration(int64(t.punchTs) + t.pn.dt + t.pn.ddt*int64(time.Since(t.pn.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
+		ts := time.Duration(int64(t.punchTs) + GNetwork.dt + GNetwork.ddtma*int64(time.Since(GNetwork.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
+		if ts > PunchTsDelay || ts < 0 {
+			ts = PunchTsDelay
+		}
 		gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
 		time.Sleep(ts)
 	}
-	gLog.Println(LvDEBUG, "handshake to ", t.config.PeerNode)
+	gLog.Println(LvDEBUG, "handshake to ", t.config.LogPeerNode())
 	var err error
-	// TODO: handle NATNone, nodes with public ip has no punching
-	if t.pn.config.natType == NATCone && t.config.peerNatType == NATCone {
+	if gConf.Network.natType == NATCone && t.config.peerNatType == NATCone {
 		err = handshakeC2C(t)
-	} else if t.config.peerNatType == NATSymmetric && t.pn.config.natType == NATSymmetric {
+	} else if t.config.peerNatType == NATSymmetric && gConf.Network.natType == NATSymmetric {
 		err = ErrorS2S
 		t.close()
-	} else if t.config.peerNatType == NATSymmetric && t.pn.config.natType == NATCone {
+	} else if t.config.peerNatType == NATSymmetric && gConf.Network.natType == NATCone {
 		err = handshakeC2S(t)
-	} else if t.config.peerNatType == NATCone && t.pn.config.natType == NATSymmetric {
+	} else if t.config.peerNatType == NATCone && gConf.Network.natType == NATSymmetric {
 		err = handshakeS2C(t)
 	} else {
 		return errors.New("unknown error")
@@ -207,7 +222,7 @@ func (t *P2PTunnel) handshake() error {
 		gLog.Println(LvERROR, "punch handshake error:", err)
 		return err
 	}
-	gLog.Printf(LvDEBUG, "handshake to %s ok", t.config.PeerNode)
+	gLog.Printf(LvDEBUG, "handshake to %s ok", t.config.LogPeerNode())
 	return nil
 }
 
@@ -216,11 +231,17 @@ func (t *P2PTunnel) connectUnderlay() (err error) {
 	case LinkModeTCP6:
 		t.conn, err = t.connectUnderlayTCP6()
 	case LinkModeTCP4:
-		t.conn, err = t.connectUnderlayTCP() // TODO: can not listen the same tcp port in pararell
+		t.conn, err = t.connectUnderlayTCP()
 	case LinkModeTCPPunch:
+		if gConf.Network.natType == NATSymmetric || t.config.peerNatType == NATSymmetric {
+			t.conn, err = t.connectUnderlayTCPSymmetric()
+		} else {
+			t.conn, err = t.connectUnderlayTCP()
+		}
+	case LinkModeIntranet:
 		t.conn, err = t.connectUnderlayTCP()
 	case LinkModeUDPPunch:
-		t.conn, err = t.connectUnderlayQuic()
+		t.conn, err = t.connectUnderlayUDP()
 
 	}
 	if err != nil {
@@ -231,59 +252,70 @@ func (t *P2PTunnel) connectUnderlay() (err error) {
 	}
 	t.setRun(true)
 	go t.readLoop()
-	go t.heartbeatLoop()
+	go t.writeLoop()
 	return nil
 }
 
-func (t *P2PTunnel) connectUnderlayQuic() (c underlay, err error) {
-	gLog.Println(LvINFO, "connectUnderlayQuic start")
-	defer gLog.Println(LvINFO, "connectUnderlayQuic end")
-	var qConn *underlayQUIC
+func (t *P2PTunnel) connectUnderlayUDP() (c underlay, err error) {
+	gLog.Printf(LvDEBUG, "connectUnderlayUDP %s start ", t.config.LogPeerNode())
+	defer gLog.Printf(LvDEBUG, "connectUnderlayUDP %s end ", t.config.LogPeerNode())
+	var ul underlay
+	underlayProtocol := t.config.UnderlayProtocol
+	if underlayProtocol == "" {
+		underlayProtocol = "quic"
+	}
 	if t.config.isUnderlayServer == 1 {
 		time.Sleep(time.Millisecond * 10) // punching udp port will need some times in some env
-		qConn, err = listenQuic(t.la.String(), TunnelIdleTimeout)
-		if err != nil {
-			gLog.Println(LvINFO, "listen quic error:", err, ", retry...")
+		go GNetwork.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+		if t.config.UnderlayProtocol == "kcp" {
+			ul, err = listenKCP(t.localHoleAddr.String(), TunnelIdleTimeout)
+		} else {
+			ul, err = listenQuic(t.localHoleAddr.String(), TunnelIdleTimeout)
 		}
-		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-		err = qConn.Accept()
+
 		if err != nil {
-			qConn.CloseListener()
-			return nil, fmt.Errorf("accept quic error:%s", err)
+			gLog.Printf(LvINFO, "listen %s error:%s", underlayProtocol, err)
+			return nil, err
 		}
-		_, buff, err := qConn.ReadBuffer()
+
+		_, buff, err := ul.ReadBuffer()
 		if err != nil {
-			qConn.listener.Close()
+			ul.Close()
 			return nil, fmt.Errorf("read start msg error:%s", err)
 		}
 		if buff != nil {
 			gLog.Println(LvDEBUG, string(buff))
 		}
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
-		gLog.Println(LvDEBUG, "quic connection ok")
-		return qConn, nil
+		ul.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
+		gLog.Printf(LvDEBUG, "%s connection ok", underlayProtocol)
+		return ul, nil
 	}
 
 	//else
-	conn, e := net.ListenUDP("udp", t.la)
-	if e != nil {
+	conn, errL := net.ListenUDP("udp", t.localHoleAddr)
+	if errL != nil {
 		time.Sleep(time.Millisecond * 10)
-		conn, e = net.ListenUDP("udp", t.la)
-		if e != nil {
-			return nil, fmt.Errorf("quic listen error:%s", e)
+		conn, errL = net.ListenUDP("udp", t.localHoleAddr)
+		if errL != nil {
+			return nil, fmt.Errorf("%s listen error:%s", underlayProtocol, errL)
 		}
 	}
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, HandshakeTimeout*3)
-	gLog.Println(LvDEBUG, "quic dial to ", t.ra.String())
-	qConn, e = dialQuic(conn, t.ra, TunnelIdleTimeout)
-	if e != nil {
-		return nil, fmt.Errorf("quic dial to %s error:%s", t.ra.String(), e)
+	GNetwork.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
+	gLog.Printf(LvDEBUG, "%s dial to %s", underlayProtocol, t.remoteHoleAddr.String())
+	if t.config.UnderlayProtocol == "kcp" {
+		ul, errL = dialKCP(conn, t.remoteHoleAddr, TunnelIdleTimeout)
+	} else {
+		ul, errL = dialQuic(conn, t.remoteHoleAddr, TunnelIdleTimeout)
+	}
+
+	if errL != nil {
+		return nil, fmt.Errorf("%s dial to %s error:%s", underlayProtocol, t.remoteHoleAddr.String(), errL)
 	}
 	handshakeBegin := time.Now()
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
-	_, buff, err := qConn.ReadBuffer()
-	if e != nil {
-		qConn.listener.Close()
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
+	_, buff, err := ul.ReadBuffer() // TODO: kcp need timeout
+	if err != nil {
+		ul.Close()
 		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
 	}
 	if buff != nil {
@@ -291,120 +323,218 @@ func (t *P2PTunnel) connectUnderlayQuic() (c underlay, err error) {
 	}
 
 	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
-	gLog.Println(LvDEBUG, "quic connection ok")
+	gLog.Printf(LvINFO, "%s connection ok", underlayProtocol)
 	t.linkModeWeb = LinkModeUDPPunch
-	return qConn, nil
+	return ul, nil
 }
 
-// websocket
 func (t *P2PTunnel) connectUnderlayTCP() (c underlay, err error) {
-	gLog.Println(LvINFO, "connectUnderlayTCP start")
-	defer gLog.Println(LvINFO, "connectUnderlayTCP end")
-	var qConn *underlayTCP
+	gLog.Printf(LvDEBUG, "connectUnderlayTCP %s start ", t.config.LogPeerNode())
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCP %s end ", t.config.LogPeerNode())
+	var ul *underlayTCP
+	peerIP := t.config.peerIP
+	if t.config.linkMode == LinkModeIntranet {
+		peerIP = t.config.peerLanIP
+	}
+	// server side
 	if t.config.isUnderlayServer == 1 {
-		qConn, err = listenTCP(t.config.peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode, t)
+		ul, err = listenTCP(peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode, t)
 		if err != nil {
 			return nil, fmt.Errorf("listen TCP error:%s", err)
 		}
-
-		_, buff, err := qConn.ReadBuffer()
-		if err != nil {
-			return nil, fmt.Errorf("read start msg error:%s", err)
-		}
-		if buff != nil {
-			gLog.Println(LvDEBUG, string(buff))
-		}
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
 		gLog.Println(LvINFO, "TCP connection ok")
-		return qConn, nil
+		t.linkModeWeb = LinkModeIPv4
+		if t.config.linkMode == LinkModeIntranet {
+			t.linkModeWeb = LinkModeIntranet
+		}
+		return ul, nil
 	}
 
 	// client side
 	if t.config.linkMode == LinkModeTCP4 {
-		t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, HandshakeTimeout*3)
+		GNetwork.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
 	} else { //tcp punch should sleep for punch the same time
-		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) == LESS {
+		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
 			gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
 		} else {
-			ts := time.Duration(int64(t.punchTs) + t.pn.dt + t.pn.ddt*int64(time.Since(t.pn.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
+			ts := time.Duration(int64(t.punchTs) + GNetwork.dt + GNetwork.ddtma*int64(time.Since(GNetwork.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
+			if ts > PunchTsDelay || ts < 0 {
+				ts = PunchTsDelay
+			}
 			gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
 			time.Sleep(ts)
 		}
 	}
-
-	gLog.Println(LvDEBUG, (time.Now().UnixNano()-t.pn.dt)/(int64)(time.Millisecond), " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", t.coneLocalPort), "-->", fmt.Sprintf("%s:%d", t.config.peerIP, t.config.peerConeNatPort))
-	qConn, err = dialTCP(t.config.peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode)
+	ul, err = dialTCP(peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode)
 	if err != nil {
 		return nil, fmt.Errorf("TCP dial to %s:%d error:%s", t.config.peerIP, t.config.peerConeNatPort, err)
 	}
 	handshakeBegin := time.Now()
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
-	_, buff, err := qConn.ReadBuffer()
+	tidBuff := new(bytes.Buffer)
+	binary.Write(tidBuff, binary.LittleEndian, t.id)
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, tidBuff.Bytes()) //  tunnelID
+	_, buff, err := ul.ReadBuffer()
 	if err != nil {
 		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
 	}
 	if buff != nil {
-		gLog.Println(LvDEBUG, string(buff))
+		gLog.Println(LvDEBUG, "hello ", string(buff))
 	}
 
 	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
 	gLog.Println(LvINFO, "TCP connection ok")
 	t.linkModeWeb = LinkModeIPv4
-	return qConn, nil
+	if t.config.linkMode == LinkModeIntranet {
+		t.linkModeWeb = LinkModeIntranet
+	}
+	return ul, nil
+}
+
+func (t *P2PTunnel) connectUnderlayTCPSymmetric() (c underlay, err error) {
+	gLog.Printf(LvDEBUG, "connectUnderlayTCPSymmetric %s start ", t.config.LogPeerNode())
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCPSymmetric %s end ", t.config.LogPeerNode())
+	ts := time.Duration(int64(t.punchTs) + GNetwork.dt + GNetwork.ddtma*int64(time.Since(GNetwork.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
+	if ts > PunchTsDelay || ts < 0 {
+		ts = PunchTsDelay
+	}
+	gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
+	time.Sleep(ts)
+	startTime := time.Now()
+	t.linkModeWeb = LinkModeTCPPunch
+	gotCh := make(chan *underlayTCP, 1)
+	var wg sync.WaitGroup
+	var success atomic.Int32
+	if t.config.peerNatType == NATSymmetric { // c2s
+		randPorts := rand.Perm(65532)
+		for i := 0; i < SymmetricHandshakeNum; i++ {
+			wg.Add(1)
+			go func(port int) {
+				defer wg.Done()
+				ul, err := dialTCP(t.config.peerIP, port, t.coneLocalPort, LinkModeTCPPunch)
+				if err != nil {
+					return
+				}
+				if !success.CompareAndSwap(0, 1) {
+					ul.Close() // only cone side close
+					return
+				}
+				err = ul.WriteMessage(MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+				if err != nil {
+					ul.Close()
+					return
+				}
+				_, buff, err := ul.ReadBuffer()
+				if err != nil {
+					gLog.Println(LvDEBUG, "c2s ul.ReadBuffer error:", err)
+					return
+				}
+				req := P2PHandshakeReq{}
+				if err = json.Unmarshal(buff, &req); err != nil {
+					return
+				}
+				if req.ID != t.id {
+					return
+				}
+				gLog.Printf(LvINFO, "handshakeS2C TCP ok. cost %dms", time.Since(startTime)/time.Millisecond)
+
+				gotCh <- ul
+				close(gotCh)
+			}(randPorts[i] + 2)
+		}
+
+	} else { // s2c
+		for i := 0; i < SymmetricHandshakeNum; i++ {
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+				ul, err := dialTCP(t.config.peerIP, t.config.peerConeNatPort, 0, LinkModeTCPPunch)
+				if err != nil {
+					return
+				}
+
+				_, buff, err := ul.ReadBuffer()
+				if err != nil {
+					gLog.Println(LvDEBUG, "s2c ul.ReadBuffer error:", err)
+					return
+				}
+				req := P2PHandshakeReq{}
+				if err = json.Unmarshal(buff, &req); err != nil {
+					return
+				}
+				if req.ID != t.id {
+					return
+				}
+				err = ul.WriteMessage(MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+				if err != nil {
+					ul.Close()
+					return
+				}
+				if success.CompareAndSwap(0, 1) {
+					gotCh <- ul
+					close(gotCh)
+				}
+			}()
+		}
+	}
+	select {
+	case <-time.After(HandshakeTimeout):
+		return nil, fmt.Errorf("wait tcp handshake timeout")
+	case ul := <-gotCh:
+		return ul, nil
+	}
 }
 
 func (t *P2PTunnel) connectUnderlayTCP6() (c underlay, err error) {
-	gLog.Println(LvINFO, "connectUnderlayTCP6 start")
-	defer gLog.Println(LvINFO, "connectUnderlayTCP6 end")
-	var qConn *underlayTCP6
+	gLog.Printf(LvDEBUG, "connectUnderlayTCP6 %s start ", t.config.LogPeerNode())
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCP6 %s end ", t.config.LogPeerNode())
+	var ul *underlayTCP6
 	if t.config.isUnderlayServer == 1 {
-		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-		qConn, err = listenTCP6(t.coneNatPort, HandshakeTimeout)
+		GNetwork.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+		ul, err = listenTCP6(t.coneNatPort, UnderlayConnectTimeout)
 		if err != nil {
 			return nil, fmt.Errorf("listen TCP6 error:%s", err)
 		}
-		_, buff, err := qConn.ReadBuffer()
+		_, buff, err := ul.ReadBuffer()
 		if err != nil {
-			qConn.listener.Close()
 			return nil, fmt.Errorf("read start msg error:%s", err)
 		}
 		if buff != nil {
 			gLog.Println(LvDEBUG, string(buff))
 		}
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
+		ul.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
 		gLog.Println(LvDEBUG, "TCP6 connection ok")
-		return qConn, nil
+		t.linkModeWeb = LinkModeIPv6
+		return ul, nil
 	}
 
 	//else
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, HandshakeTimeout*3)
+	GNetwork.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
 	gLog.Println(LvDEBUG, "TCP6 dial to ", t.config.peerIPv6)
-	qConn, err = dialTCP6(t.config.peerIPv6, t.config.peerConeNatPort)
-	if err != nil {
+	ul, err = dialTCP6(t.config.peerIPv6, t.config.peerConeNatPort)
+	if err != nil || ul == nil {
 		return nil, fmt.Errorf("TCP6 dial to %s:%d error:%s", t.config.peerIPv6, t.config.peerConeNatPort, err)
 	}
 	handshakeBegin := time.Now()
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
-	_, buff, err := qConn.ReadBuffer()
-	if err != nil {
-		qConn.listener.Close()
-		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
+	_, buff, errR := ul.ReadBuffer()
+	if errR != nil {
+		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", errR)
 	}
 	if buff != nil {
 		gLog.Println(LvDEBUG, string(buff))
 	}
 
 	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
-	gLog.Println(LvDEBUG, "TCP6 connection ok")
+	gLog.Println(LvINFO, "TCP6 connection ok")
 	t.linkModeWeb = LinkModeIPv6
-	return qConn, nil
+	return ul, nil
 }
 
 func (t *P2PTunnel) readLoop() {
 	decryptData := make([]byte, ReadBuffLen+PaddingSize) // 16 bytes for padding
 	gLog.Printf(LvDEBUG, "%d tunnel readloop start", t.id)
 	for t.isRuning() {
-		t.conn.SetReadDeadline(time.Now().Add(TunnelIdleTimeout))
+		t.conn.SetReadDeadline(time.Now().Add(TunnelHeartbeatTime * 2))
 		head, body, err := t.conn.ReadBuffer()
 		if err != nil {
 			if t.isRuning() {
@@ -413,24 +543,29 @@ func (t *P2PTunnel) readLoop() {
 			break
 		}
 		if head.MainType != MsgP2P {
+			gLog.Printf(LvWARN, "%d head.MainType != MsgP2P", t.id)
 			continue
 		}
+		// TODO: replace some case implement to functions
 		switch head.SubType {
 		case MsgTunnelHeartbeat:
+			t.hbMtx.Lock()
 			t.hbTime = time.Now()
+			t.hbMtx.Unlock()
 			t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeatAck, nil)
-			gLog.Printf(LvDEBUG, "%d read tunnel heartbeat", t.id)
+			gLog.Printf(LvDev, "%d read tunnel heartbeat", t.id)
 		case MsgTunnelHeartbeatAck:
 			t.hbMtx.Lock()
 			t.hbTime = time.Now()
 			t.hbMtx.Unlock()
-			gLog.Printf(LvDEBUG, "%d read tunnel heartbeat ack", t.id)
+			gLog.Printf(LvDev, "%d read tunnel heartbeat ack", t.id)
 		case MsgOverlayData:
 			if len(body) < overlayHeaderSize {
+				gLog.Printf(LvWARN, "%d len(body) < overlayHeaderSize", t.id)
 				continue
 			}
 			overlayID := binary.LittleEndian.Uint64(body[:8])
-			gLog.Printf(LvDEBUG, "%d tunnel read overlay data %d bodylen=%d", t.id, overlayID, head.DataLen)
+			gLog.Printf(LvDev, "%d tunnel read overlay data %d bodylen=%d", t.id, overlayID, head.DataLen)
 			s, ok := t.overlayConns.Load(overlayID)
 			if !ok {
 				// debug level, when overlay connection closed, always has some packet not found tunnel
@@ -450,25 +585,31 @@ func (t *P2PTunnel) readLoop() {
 			if err != nil {
 				gLog.Println(LvERROR, "overlay write error:", err)
 			}
+		case MsgNodeData:
+			t.handleNodeData(head, body, false)
+		case MsgRelayNodeData:
+			t.handleNodeData(head, body, true)
 		case MsgRelayData:
-			gLog.Printf(LvDEBUG, "got relay data datalen=%d", head.DataLen)
 			if len(body) < 8 {
 				continue
 			}
 			tunnelID := binary.LittleEndian.Uint64(body[:8])
-			t.pn.relay(tunnelID, body[8:])
+			gLog.Printf(LvDev, "relay data to %d, len=%d", tunnelID, head.DataLen-RelayHeaderSize)
+			if err := GNetwork.relay(tunnelID, body[RelayHeaderSize:]); err != nil {
+				gLog.Printf(LvERROR, "%s:%d relay to %d len=%d error:%s", t.config.LogPeerNode(), t.id, tunnelID, len(body), ErrRelayTunnelNotFound)
+			}
 		case MsgRelayHeartbeat:
 			req := RelayHeartbeat{}
 			if err := json.Unmarshal(body, &req); err != nil {
 				gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
 				continue
 			}
-			gLog.Printf(LvDEBUG, "got MsgRelayHeartbeat from %d:%d", req.RelayTunnelID, req.AppID)
-			relayHead := new(bytes.Buffer)
-			binary.Write(relayHead, binary.LittleEndian, req.RelayTunnelID)
-			msg, _ := newMessage(MsgP2P, MsgRelayHeartbeatAck, &req)
-			msgWithHead := append(relayHead.Bytes(), msg...)
-			t.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+			// TODO: debug relay heartbeat
+			gLog.Printf(LvDEBUG, "read MsgRelayHeartbeat from rtid:%d,appid:%d", req.RelayTunnelID, req.AppID)
+			// update app hbtime
+			GNetwork.updateAppHeartbeat(req.AppID)
+			req.From = gConf.Network.Node
+			t.WriteMessage(req.RelayTunnelID, MsgP2P, MsgRelayHeartbeatAck, &req)
 		case MsgRelayHeartbeatAck:
 			req := RelayHeartbeat{}
 			err := json.Unmarshal(body, &req)
@@ -476,8 +617,9 @@ func (t *P2PTunnel) readLoop() {
 				gLog.Printf(LvERROR, "wrong RelayHeartbeat:%s", err)
 				continue
 			}
-			gLog.Printf(LvDEBUG, "got MsgRelayHeartbeatAck to %d", req.AppID)
-			t.pn.updateAppHeartbeat(req.AppID)
+			// TODO: debug relay heartbeat
+			gLog.Printf(LvDEBUG, "read MsgRelayHeartbeatAck to appid:%d", req.AppID)
+			GNetwork.updateAppHeartbeat(req.AppID)
 		case MsgOverlayConnectReq:
 			req := OverlayConnectReq{}
 			if err := json.Unmarshal(body, &req); err != nil {
@@ -485,7 +627,7 @@ func (t *P2PTunnel) readLoop() {
 				continue
 			}
 			// app connect only accept token(not relay totp token), avoid someone using the share relay node's token
-			if req.Token != t.pn.config.Token {
+			if req.Token != gConf.Network.Token {
 				gLog.Println(LvERROR, "Access Denied:", req.Token)
 				continue
 			}
@@ -504,7 +646,7 @@ func (t *P2PTunnel) readLoop() {
 			if req.Protocol == "udp" {
 				oConn.connUDP, err = net.DialUDP("udp", nil, &net.UDPAddr{IP: net.ParseIP(req.DstIP), Port: req.DstPort})
 			} else {
-				oConn.connTCP, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.DstIP, req.DstPort), HandshakeTimeout)
+				oConn.connTCP, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.DstIP, req.DstPort), ReadMsgTimeout)
 
 			}
 			if err != nil {
@@ -538,27 +680,40 @@ func (t *P2PTunnel) readLoop() {
 		default:
 		}
 	}
-	t.setRun(false)
-	t.conn.Close()
+	t.close()
 	gLog.Printf(LvDEBUG, "%d tunnel readloop end", t.id)
 }
 
-func (t *P2PTunnel) heartbeatLoop() {
+func (t *P2PTunnel) writeLoop() {
+	t.hbMtx.Lock()
+	t.hbTime = time.Now() // init
+	t.hbMtx.Unlock()
 	tc := time.NewTicker(TunnelHeartbeatTime)
 	defer tc.Stop()
-	gLog.Printf(LvDEBUG, "%d tunnel heartbeatLoop start", t.id)
-	defer gLog.Printf(LvDEBUG, "%d tunnel heartbeatLoop end", t.id)
+	gLog.Printf(LvDEBUG, "%s:%d tunnel writeLoop start", t.config.LogPeerNode(), t.id)
+	defer gLog.Printf(LvDEBUG, "%s:%d tunnel writeLoop end", t.config.LogPeerNode(), t.id)
 	for t.isRuning() {
 		select {
+		case buff := <-t.writeDataSmall:
+			t.conn.WriteBuffer(buff)
+			// gLog.Printf(LvDEBUG, "write icmp %d", time.Now().Unix())
+		default:
+			select {
+			case buff := <-t.writeDataSmall:
+				t.conn.WriteBuffer(buff)
+				// gLog.Printf(LvDEBUG, "write icmp %d", time.Now().Unix())
+			case buff := <-t.writeData:
+				t.conn.WriteBuffer(buff)
 			case <-tc.C:
 				// tunnel send
 				err := t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeat, nil)
 				if err != nil {
 					gLog.Printf(LvERROR, "%d write tunnel heartbeat error %s", t.id, err)
-				t.setRun(false)
+					t.close()
 					return
 				}
-			gLog.Printf(LvDEBUG, "%d write tunnel heartbeat ok", t.id)
+				gLog.Printf(LvDev, "%d write tunnel heartbeat ok", t.id)
+			}
 		}
 	}
 }
@@ -569,24 +724,24 @@ func (t *P2PTunnel) listen() error {
 		Error:   0,
 		Detail:  "connect ok",
 		To:      t.config.PeerNode,
-		From:    t.pn.config.Node,
-		NatType: t.pn.config.natType,
-		HasIPv4: t.pn.config.hasIPv4,
-		// IPv6:            t.pn.config.IPv6,
-		HasUPNPorNATPMP: t.pn.config.hasUPNPorNATPMP,
-		FromIP:          t.pn.config.publicIP,
+		From:    gConf.Network.Node,
+		NatType: gConf.Network.natType,
+		HasIPv4: gConf.Network.hasIPv4,
+		// IPv6:            gConf.Network.IPv6,
+		HasUPNPorNATPMP: gConf.Network.hasUPNPorNATPMP,
+		FromIP:          gConf.Network.publicIP,
 		ConeNatPort:     t.coneNatPort,
 		ID:              t.id,
-		PunchTs:         uint64(time.Now().UnixNano() + int64(PunchTsDelay) - t.pn.dt),
+		PunchTs:         uint64(time.Now().UnixNano() + int64(PunchTsDelay) - GNetwork.dt),
 		Version:         OpenP2PVersion,
 	}
 	t.punchTs = rsp.PunchTs
 	// only private node set ipv6
-	if t.config.fromToken == t.pn.config.Token {
+	if t.config.fromToken == gConf.Network.Token {
 		rsp.IPv6 = gConf.IPv6()
 	}
 
-	t.pn.push(t.config.PeerNode, MsgPushConnectRsp, rsp)
+	GNetwork.push(t.config.PeerNode, MsgPushConnectRsp, rsp)
 	gLog.Printf(LvDEBUG, "p2ptunnel wait for connecting")
 	t.tunnelServer = true
 	return t.start()
@@ -601,3 +756,50 @@ func (t *P2PTunnel) closeOverlayConns(appID uint64) {
 		return true
 	})
 }
+
+func (t *P2PTunnel) handleNodeData(head *openP2PHeader, body []byte, isRelay bool) {
+	gLog.Printf(LvDev, "%d tunnel read node data bodylen=%d, relay=%t", t.id, head.DataLen, isRelay)
+	ch := GNetwork.nodeData
+	// if body[9] == 1 { // TODO: deal relay
+	// 	ch = GNetwork.nodeDataSmall
+	// 	gLog.Printf(LvDEBUG, "read icmp %d", time.Now().Unix())
+	// }
+	if isRelay {
+		fromPeerID := binary.LittleEndian.Uint64(body[:8])
+		ch <- &NodeData{fromPeerID, body[8:]} // TODO: cache peerNodeID; encrypt/decrypt
+	} else {
+		ch <- &NodeData{NodeNameToID(t.config.PeerNode), body} // TODO: cache peerNodeID; encrypt/decrypt
+	}
+}
+
+func (t *P2PTunnel) asyncWriteNodeData(mainType, subType uint16, data []byte) {
+	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	// if len(data) < 192 {
+	if data[9] == 1 { // icmp
+		select {
+		case t.writeDataSmall <- writeBytes:
+			// gLog.Printf(LvWARN, "%s:%d t.writeDataSmall write %d", t.config.PeerNode, t.id, len(t.writeDataSmall))
+		default:
+			gLog.Printf(LvWARN, "%s:%d t.writeDataSmall is full, drop it", t.config.LogPeerNode(), t.id)
+		}
+	} else {
+		select {
+		case t.writeData <- writeBytes:
+		default:
+			gLog.Printf(LvWARN, "%s:%d t.writeData is full, drop it", t.config.LogPeerNode(), t.id)
+		}
+	}
+
+}
+
+func (t *P2PTunnel) WriteMessage(rtid uint64, mainType uint16, subType uint16, req interface{}) error {
+	if rtid == 0 {
+		return t.conn.WriteMessage(mainType, subType, &req)
+	}
+	relayHead := new(bytes.Buffer)
+	binary.Write(relayHead, binary.LittleEndian, rtid)
+	msg, _ := newMessage(mainType, subType, &req)
+	msgWithHead := append(relayHead.Bytes(), msg...)
+	return t.conn.WriteBytes(mainType, MsgRelayData, msgWithHead)
+
+}

core/p2ptunnel_test.go

@@ -0,0 +1,24 @@
+package openp2p
+
+import (
+	"fmt"
+	"testing"
+)
+
+func TestSelectPriority(t *testing.T) {
+	writeData := make(chan []byte, WriteDataChanSize)
+	writeDataSmall := make(chan []byte, WriteDataChanSize/30)
+	for i := 0; i < 100; i++ {
+		writeData <- []byte("data")
+		writeDataSmall <- []byte("small data")
+	}
+	for i := 0; i < 100; i++ {
+		select {
+		case buff := <-writeDataSmall:
+			fmt.Printf("got small data:%s\n", string(buff))
+		case buff := <-writeData:
+			fmt.Printf("got data:%s\n", string(buff))
+		}
+	}
+
+}

core/ping.go

@@ -0,0 +1,87 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"os"
+	"time"
+
+	"golang.org/x/net/icmp"
+	"golang.org/x/net/ipv4"
+)
+
+// 定义ICMP回显请求和应答的结构
+type ICMPMessage struct {
+	Type     uint8
+	Code     uint8
+	Checksum uint16
+	Ident    uint16
+	Seq      uint16
+	Data     []byte
+}
+
+// Ping sends an ICMP Echo request to the specified host and returns the response time.
+func Ping(host string) (time.Duration, error) {
+	// Resolve the IP address of the host
+	ipAddr, err := net.ResolveIPAddr("ip4", host)
+	if err != nil {
+		return 0, fmt.Errorf("failed to resolve host: %v", err)
+	}
+
+	// Create an ICMP listener
+	conn, err := net.ListenPacket("ip4:icmp", "0.0.0.0")
+	if err != nil {
+		return 0, fmt.Errorf("failed to create ICMP connection: %v", err)
+	}
+	defer conn.Close()
+
+	// Create an ICMP Echo request message
+	message := icmp.Message{
+		Type: ipv4.ICMPTypeEcho,
+		Code: 0,
+		Body: &icmp.Echo{
+			ID:   os.Getpid() & 0xffff,
+			Seq:  1,
+			Data: []byte("HELLO-R-U-THERE"),
+		},
+	}
+
+	// Marshal the message into binary form
+	messageBytes, err := message.Marshal(nil)
+	if err != nil {
+		return 0, fmt.Errorf("failed to marshal ICMP message: %v", err)
+	}
+
+	// Send the ICMP Echo request
+	start := time.Now()
+	if _, err := conn.WriteTo(messageBytes, ipAddr); err != nil {
+		return 0, fmt.Errorf("failed to send ICMP request: %v", err)
+	}
+
+	// Set a deadline for the response
+	err = conn.SetReadDeadline(time.Now().Add(3 * time.Second))
+	if err != nil {
+		return 0, fmt.Errorf("failed to set read deadline: %v", err)
+	}
+
+	// Read the ICMP response
+	response := make([]byte, 1500)
+	n, _, err := conn.ReadFrom(response)
+	if err != nil {
+		return 0, fmt.Errorf("failed to read ICMP response: %v", err)
+	}
+
+	// Parse the ICMP response message
+	parsedMessage, err := icmp.ParseMessage(ipv4.ICMPTypeEchoReply.Protocol(), response[:n])
+	if err != nil {
+		return 0, fmt.Errorf("failed to parse ICMP response: %v", err)
+	}
+
+	// Check if the response is an Echo reply
+	if parsedMessage.Type == ipv4.ICMPTypeEchoReply {
+		duration := time.Since(start)
+		return duration, nil
+	} else {
+		return 0, fmt.Errorf("unexpected ICMP message: %+v", parsedMessage)
+	}
+}

core/protocol.go

@@ -10,11 +10,14 @@ import (
 	"time"
 )
 
-const OpenP2PVersion = "3.10.9"
+const OpenP2PVersion = "3.21.12"
 const ProductName string = "openp2p"
 const LeastSupportVersion = "3.0.0"
 const SyncServerTimeVersion = "3.9.0"
 const SymmetricSimultaneouslySendVersion = "3.10.7"
+const PublicIPVersion = "3.11.2"
+const SupportIntranetVersion = "3.14.5"
+const SupportDualTunnelVersion = "3.15.5"
 
 const (
 	IfconfigPort1 = 27180
@@ -39,6 +42,8 @@ type PushHeader struct {
 
 var PushHeaderSize = binary.Size(PushHeader{})
 
+const RelayHeaderSize = 8
+
 type overlayHeader struct {
 	id uint64
 }
@@ -79,6 +84,7 @@ const (
 	MsgRelay     = 5
 	MsgReport    = 6
 	MsgQuery     = 7
+	MsgSDWAN     = 8
 )
 
 // TODO: seperate node push and web push.
@@ -99,6 +105,10 @@ const (
 	MsgPushAPPKey               = 13
 	MsgPushReportLog            = 14
 	MsgPushDstNodeOnline        = 15
+	MsgPushReportGoroutine      = 16
+	MsgPushReportMemApps        = 17
+	MsgPushServerSideSaveMemApp = 18
+	MsgPushCheckRemoteService   = 19
 )
 
 // MsgP2P sub type message
@@ -116,6 +126,8 @@ const (
 	MsgRelayData
 	MsgRelayHeartbeat
 	MsgRelayHeartbeatAck
+	MsgNodeData
+	MsgRelayNodeData
 )
 
 // MsgRelay sub type message
@@ -131,30 +143,40 @@ const (
 	MsgReportConnect
 	MsgReportApps
 	MsgReportLog
+	MsgReportMemApps
+	MsgReportResponse
 )
 
 const (
 	ReadBuffLen               = 4096 // for UDP maybe not enough
-	NetworkHeartbeatTime  = time.Second * 30 // TODO: server no response hb, save flow
+	NetworkHeartbeatTime      = time.Second * 30
 	TunnelHeartbeatTime       = time.Second * 10 // some nat udp session expired time less than 15s. change to 10s
+	UnderlayTCPKeepalive      = time.Second * 5
+	UnderlayTCPConnectTimeout = time.Second * 5
 	TunnelIdleTimeout         = time.Minute
 	SymmetricHandshakeNum     = 800 // 0.992379
 	// SymmetricHandshakeNum        = 1000 // 0.999510
 	SymmetricHandshakeInterval = time.Millisecond
-	HandshakeTimeout           = time.Second * 10
-	PeerAddRelayTimeount       = time.Second * 30 // peer need times
+	HandshakeTimeout           = time.Second * 7
+	PunchTsDelay               = time.Second * 3
+	PeerAddRelayTimeount       = time.Second * 30 // peer need times. S2C\TCP\TCP Punch\UDP Punch
 	CheckActiveTimeout         = time.Second * 5
+	ReadMsgTimeout             = time.Second * 5
 	PaddingSize                = 16
 	AESKeySize                 = 16
 	MaxRetry                   = 10
-	Cone2ConePunchMaxRetry     = 1
-	PublicIPEchoTimeout        = time.Second * 1
+	Cone2ConeTCPPunchMaxRetry  = 1
+	Cone2ConeUDPPunchMaxRetry  = 1
+	PublicIPEchoTimeout        = time.Second * 3
 	NatTestTimeout             = time.Second * 5
 	UDPReadTimeout             = time.Second * 5
 	ClientAPITimeout           = time.Second * 10
 	UnderlayConnectTimeout     = time.Second * 10
 	MaxDirectTry               = 3
-	PunchTsDelay               = time.Second * 3
+
+	// sdwan
+	ReadTunBuffSize = 1600
+	ReadTunBuffNum  = 10
 )
 
 // NATNone has public ip
@@ -177,6 +199,7 @@ const (
 	LinkModeUDPPunch = "udppunch"
 	LinkModeTCPPunch = "tcppunch"
 	LinkModeIPv4     = "ipv4"     // for web
+	LinkModeIntranet = "intranet" // for web
 	LinkModeIPv6     = "ipv6"     // for web
 	LinkModeTCP6     = "tcp6"
 	LinkModeTCP4     = "tcp4"
@@ -189,6 +212,17 @@ const (
 	MsgQueryPeerInfoRsp
 )
 
+const (
+	MsgSDWANInfoReq = iota
+	MsgSDWANInfoRsp
+)
+
+// MsgNATDetect
+const (
+	MsgNAT = iota
+	MsgPublicIP
+)
+
 func newMessage(mainType uint16, subType uint16, packet interface{}) ([]byte, error) {
 	data, err := json.Marshal(packet)
 	if err != nil {
@@ -209,7 +243,7 @@ func newMessage(mainType uint16, subType uint16, packet interface{}) ([]byte, er
 	return writeBytes, nil
 }
 
-func nodeNameToID(name string) uint64 {
+func NodeNameToID(name string) uint64 {
 	return crc64.Checksum([]byte(name), crc64.MakeTable(crc64.ISO))
 }
 
@@ -228,6 +262,7 @@ type PushConnectReq struct {
 	AppKey           uint64 `json:"appKey,omitempty"` // for underlay tcp
 	LinkMode         string `json:"linkMode,omitempty"`
 	IsUnderlayServer int    `json:"isServer,omitempty"`         // Requset spec peer is server
+	UnderlayProtocol string `json:"underlayProtocol,omitempty"` // quic or kcp, default quic
 }
 type PushDstNodeOnline struct {
 	Node string `json:"node,omitempty"`
@@ -259,6 +294,7 @@ type LoginRsp struct {
 	Node          string `json:"node,omitempty"`
 	Token         uint64 `json:"token,omitempty"`
 	Ts            int64  `json:"ts,omitempty"`
+	LoginMaxDelay int    `json:"loginMaxDelay,omitempty"` // seconds
 }
 
 type NatDetectReq struct {
@@ -305,7 +341,9 @@ type RelayNodeRsp struct {
 type AddRelayTunnelReq struct {
 	From          string `json:"from,omitempty"`
 	RelayName     string `json:"relayName,omitempty"`
+	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"`
 	RelayToken    uint64 `json:"relayToken,omitempty"`
+	RelayMode     string `json:"relayMode,omitempty"`
 	AppID         uint64 `json:"appID,omitempty"`  // deprecated
 	AppKey        uint64 `json:"appKey,omitempty"` // deprecated
 }
@@ -316,6 +354,7 @@ type APPKeySync struct {
 }
 
 type RelayHeartbeat struct {
+	From          string `json:"from,omitempty"`
 	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"`
 	AppID         uint64 `json:"appID,omitempty"`
 }
@@ -351,6 +390,7 @@ type AppInfo struct {
 	AppName        string `json:"appName,omitempty"`
 	Error          string `json:"error,omitempty"`
 	Protocol       string `json:"protocol,omitempty"`
+	PunchPriority  int    `json:"punchPriority,omitempty"`
 	Whitelist      string `json:"whitelist,omitempty"`
 	SrcPort        int    `json:"srcPort,omitempty"`
 	Protocol0      string `json:"protocol0,omitempty"`
@@ -364,6 +404,7 @@ type AppInfo struct {
 	PeerIP         string `json:"peerIP,omitempty"`
 	ShareBandwidth int    `json:"shareBandWidth,omitempty"`
 	RelayNode      string `json:"relayNode,omitempty"`
+	SpecRelayNode  string `json:"specRelayNode,omitempty"`
 	RelayMode      string `json:"relayMode,omitempty"`
 	LinkMode       string `json:"linkMode,omitempty"`
 	Version        string `json:"version,omitempty"`
@@ -433,15 +474,55 @@ type QueryPeerInfoReq struct {
 	PeerNode string `json:"peerNode,omitempty"`
 }
 type QueryPeerInfoRsp struct {
+	PeerNode        string `json:"peerNode,omitempty"`
 	Online          int    `json:"online,omitempty"`
 	Version         string `json:"version,omitempty"`
 	NatType         int    `json:"natType,omitempty"`
 	IPv4            string `json:"IPv4,omitempty"`
+	LanIP           string `json:"lanIP,omitempty"`
 	HasIPv4         int    `json:"hasIPv4,omitempty"` // has public ipv4
 	IPv6            string `json:"IPv6,omitempty"`    // if public relay node, ipv6 not set
 	HasUPNPorNATPMP int    `json:"hasUPNPorNATPMP,omitempty"`
 }
 
+type SDWANNode struct {
+	Name     string `json:"name,omitempty"`
+	IP       string `json:"ip,omitempty"`
+	Resource string `json:"resource,omitempty"`
+	Enable   int32  `json:"enable,omitempty"`
+}
+
+type SDWANInfo struct {
+	ID            uint64 `json:"id,omitempty"`
+	Name          string `json:"name,omitempty"`
+	Gateway       string `json:"gateway,omitempty"`
+	Mode          string `json:"mode,omitempty"` // default: fullmesh; central
+	CentralNode   string `json:"centralNode,omitempty"`
+	ForceRelay    int32  `json:"forceRelay,omitempty"`
+	PunchPriority int32  `json:"punchPriority,omitempty"`
+	Enable        int32  `json:"enable,omitempty"`
+	Nodes         []*SDWANNode
+}
+
+const (
+	SDWANModeFullmesh = "fullmesh"
+	SDWANModeCentral  = "central"
+)
+
+type ServerSideSaveMemApp struct {
+	From          string `json:"from,omitempty"`
+	Node          string `json:"node,omitempty"`          // for server side findtunnel, maybe relayNode
+	TunnelID      uint64 `json:"tunnelID,omitempty"`      // save in app.tunnel or app.relayTunnel
+	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"` // rtid, if not 0 relay
+	RelayMode     string `json:"relayMode,omitempty"`
+	AppID         uint64 `json:"appID,omitempty"`
+}
+
+type CheckRemoteService struct {
+	Host string `json:"host,omitempty"`
+	Port uint32 `json:"port,omitempty"`
+}
+
 const rootCA = `-----BEGIN CERTIFICATE-----
 MIIDhTCCAm0CFHm0cd8dnGCbUW/OcS56jf0gvRk7MA0GCSqGSIb3DQEBCwUAMH4x
 CzAJBgNVBAYTAkNOMQswCQYDVQQIDAJHRDETMBEGA1UECgwKb3BlbnAycC5jbjET

core/sdwan.go

@@ -0,0 +1,323 @@
+package openp2p
+
+import (
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"net"
+	"runtime"
+	"strings"
+	"sync"
+	"time"
+)
+
+type PacketHeader struct {
+	version int
+	// src     uint32
+	// prot    uint8
+	protocol byte
+	dst      uint32
+	port     uint16
+}
+
+func parseHeader(b []byte, h *PacketHeader) error {
+	if len(b) < 20 {
+		return fmt.Errorf("small packet")
+	}
+	h.version = int(b[0] >> 4)
+	h.protocol = byte(b[9])
+	if h.version == 4 {
+		h.dst = binary.BigEndian.Uint32(b[16:20])
+	} else if h.version != 6 {
+		return fmt.Errorf("unknown version in ip header:%d", h.version)
+	}
+	if h.protocol == 6 || h.protocol == 17 { // TCP or UDP
+		h.port = binary.BigEndian.Uint16(b[22:24])
+	}
+	return nil
+}
+
+type sdwanNode struct {
+	name string
+	id   uint64
+}
+
+type p2pSDWAN struct {
+	nodeName      string
+	tun           *optun
+	sysRoute      sync.Map // ip:sdwanNode
+	subnet        *net.IPNet
+	gateway       net.IP
+	virtualIP     *net.IPNet
+	internalRoute *IPTree
+}
+
+func (s *p2pSDWAN) reset() {
+	gLog.Println(LvINFO, "reset sdwan when network disconnected")
+	// clear sysroute
+	delRoutesByGateway(s.gateway.String())
+	// clear internel route
+	s.internalRoute = NewIPTree("")
+	// clear p2papp
+	for _, node := range gConf.getAddNodes() {
+		gConf.delete(AppConfig{SrcPort: 0, PeerNode: node.Name})
+	}
+
+	gConf.resetSDWAN()
+}
+func (s *p2pSDWAN) init(name string) error {
+	if gConf.getSDWAN().Gateway == "" {
+		gLog.Println(LvDEBUG, "sdwan init: not in sdwan clear all ")
+	}
+	if s.internalRoute == nil {
+		s.internalRoute = NewIPTree("")
+	}
+
+	s.nodeName = name
+	if gw, sn, err := net.ParseCIDR(gConf.getSDWAN().Gateway); err == nil { // preserve old gateway
+		s.gateway = gw
+		s.subnet = sn
+	}
+
+	for _, node := range gConf.getDelNodes() {
+		gLog.Println(LvDEBUG, "sdwan init: deal deleted node: ", node.Name)
+		gLog.Printf(LvDEBUG, "sdwan init: delRoute: %s, %s ", node.IP, s.gateway.String())
+		delRoute(node.IP, s.gateway.String())
+		s.internalRoute.Del(node.IP, node.IP)
+		ipNum, _ := inetAtoN(node.IP)
+		s.sysRoute.Delete(ipNum)
+		gConf.delete(AppConfig{SrcPort: 0, PeerNode: node.Name})
+		GNetwork.DeleteApp(AppConfig{SrcPort: 0, PeerNode: node.Name})
+		arr := strings.Split(node.Resource, ",")
+		for _, r := range arr {
+			_, ipnet, err := net.ParseCIDR(r)
+			if err != nil {
+				// fmt.Println("Error parsing CIDR:", err)
+				continue
+			}
+			if ipnet.Contains(net.ParseIP(gConf.Network.localIP)) { // local ip and resource in the same lan
+				continue
+			}
+			minIP := ipnet.IP
+			maxIP := make(net.IP, len(minIP))
+			copy(maxIP, minIP)
+			for i := range minIP {
+				maxIP[i] = minIP[i] | ^ipnet.Mask[i]
+			}
+			s.internalRoute.Del(minIP.String(), maxIP.String())
+			delRoute(ipnet.String(), s.gateway.String())
+			gLog.Printf(LvDEBUG, "sdwan init: resource delRoute: %s, %s ", ipnet.String(), s.gateway.String())
+		}
+	}
+	for _, node := range gConf.getAddNodes() {
+		gLog.Println(LvDEBUG, "sdwan init: deal add node: ", node.Name)
+		ipNet := &net.IPNet{
+			IP:   net.ParseIP(node.IP),
+			Mask: s.subnet.Mask,
+		}
+		if node.Name == s.nodeName {
+			s.virtualIP = ipNet
+			gLog.Println(LvINFO, "sdwan init: start tun ", ipNet.String())
+			err := s.StartTun()
+			if err != nil {
+				gLog.Println(LvERROR, "sdwan init: start tun error:", err)
+				return err
+			}
+			gLog.Println(LvINFO, "sdwan init: start tun ok")
+			allowTunForward()
+			gLog.Printf(LvDEBUG, "sdwan init: addRoute %s %s %s", s.subnet.String(), s.gateway.String(), s.tun.tunName)
+			addRoute(s.subnet.String(), s.gateway.String(), s.tun.tunName)
+			// addRoute("255.255.255.255/32", s.gateway.String(), s.tun.tunName) // for broadcast
+			// addRoute("224.0.0.0/4", s.gateway.String(), s.tun.tunName)        // for multicast
+			initSNATRule(s.subnet.String()) // for network resource
+			continue
+		}
+		ip, err := inetAtoN(ipNet.String())
+		if err != nil {
+			return err
+		}
+		s.sysRoute.Store(ip, &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+		s.internalRoute.AddIntIP(ip, ip, &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+	}
+	for _, node := range gConf.getAddNodes() {
+		if node.Name == s.nodeName { // not deal resource itself
+			continue
+		}
+		if len(node.Resource) > 0 {
+			gLog.Printf(LvINFO, "sdwan init: deal add node: %s resource: %s", node.Name, node.Resource)
+			arr := strings.Split(node.Resource, ",")
+			for _, r := range arr {
+				// add internal route
+				_, ipnet, err := net.ParseCIDR(r)
+				if err != nil {
+					fmt.Println("sdwan init: Error parsing CIDR:", err)
+					continue
+				}
+				if ipnet.Contains(net.ParseIP(gConf.Network.localIP)) { // local ip and resource in the same lan
+					gLog.Printf(LvDEBUG, "sdwan init: local ip %s in this resource %s, ignore", gConf.Network.localIP, ipnet.IP.String())
+					continue
+				}
+				// local net could access this single ip
+				if ipnet.Mask[0] == 255 && ipnet.Mask[1] == 255 && ipnet.Mask[2] == 255 && ipnet.Mask[3] == 255 {
+					gLog.Printf(LvDEBUG, "sdwan init: ping %s start", ipnet.IP.String())
+					if _, err := Ping(ipnet.IP.String()); err == nil {
+						gLog.Printf(LvDEBUG, "sdwan init: ping %s ok, ignore this resource", ipnet.IP.String())
+						continue
+					}
+					gLog.Printf(LvDEBUG, "sdwan init: ping %s failed", ipnet.IP.String())
+				}
+				minIP := ipnet.IP
+				maxIP := make(net.IP, len(minIP))
+				copy(maxIP, minIP)
+				for i := range minIP {
+					maxIP[i] = minIP[i] | ^ipnet.Mask[i]
+				}
+				s.internalRoute.Add(minIP.String(), maxIP.String(), &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+				// add sys route
+				gLog.Printf(LvDEBUG, "sdwan init: addRoute %s %s %s", ipnet.String(), s.gateway.String(), s.tun.tunName)
+				addRoute(ipnet.String(), s.gateway.String(), s.tun.tunName)
+			}
+		}
+	}
+	gConf.retryAllMemApp()
+	gLog.Printf(LvINFO, "sdwan init ok")
+	return nil
+}
+
+func (s *p2pSDWAN) run() {
+	s.sysRoute.Range(func(key, value interface{}) bool {
+		node := value.(*sdwanNode)
+		GNetwork.ConnectNode(node.name)
+		return true
+	})
+}
+
+func (s *p2pSDWAN) readNodeLoop() {
+	gLog.Printf(LvDEBUG, "sdwan readNodeLoop start")
+	defer gLog.Printf(LvDEBUG, "sdwan readNodeLoop end")
+	writeBuff := make([][]byte, 1)
+	for {
+		nd := GNetwork.ReadNode(time.Second * 10) // TODO: read multi packet
+		if nd == nil {
+			gLog.Printf(LvDev, "waiting for node data")
+			continue
+		}
+		head := PacketHeader{}
+		parseHeader(nd.Data, &head)
+		gLog.Printf(LvDev, "write tun dst ip=%s,len=%d", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len(nd.Data))
+		if PIHeaderSize == 0 {
+			writeBuff[0] = nd.Data
+		} else {
+			writeBuff[0] = make([]byte, PIHeaderSize+len(nd.Data))
+			copy(writeBuff[0][PIHeaderSize:], nd.Data)
+		}
+
+		len, err := s.tun.Write(writeBuff, PIHeaderSize)
+		if err != nil {
+			gLog.Printf(LvDEBUG, "write tun dst ip=%s,len=%d,error:%s", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len, err)
+		}
+	}
+}
+
+func isBroadcastOrMulticast(ipUint32 uint32, subnet *net.IPNet) bool {
+	// return ipUint32 == 0xffffffff || (byte(ipUint32) == 0xff) || (ipUint32>>28 == 0xe)
+	return ipUint32 == 0xffffffff || (ipUint32>>28 == 0xe) // 225.255.255.255/32, 224.0.0.0/4
+}
+
+func (s *p2pSDWAN) routeTunPacket(p []byte, head *PacketHeader) {
+	var node *sdwanNode
+	// v, ok := s.routes.Load(ih.dst)
+	v, ok := s.internalRoute.Load(head.dst)
+	if !ok || v == nil {
+		if isBroadcastOrMulticast(head.dst, s.subnet) {
+			gLog.Printf(LvDev, "multicast ip=%s", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String())
+			GNetwork.WriteBroadcast(p)
+		}
+		return
+	} else {
+		node = v.(*sdwanNode)
+	}
+
+	err := GNetwork.WriteNode(node.id, p)
+	if err != nil {
+		gLog.Printf(LvDev, "write packet to %s fail: %s", node.name, err)
+	}
+}
+
+func (s *p2pSDWAN) readTunLoop() {
+	gLog.Printf(LvDEBUG, "sdwan readTunLoop start")
+	defer gLog.Printf(LvDEBUG, "sdwan readTunLoop end")
+	readBuff := make([][]byte, ReadTunBuffNum)
+	for i := 0; i < ReadTunBuffNum; i++ {
+		readBuff[i] = make([]byte, ReadTunBuffSize+PIHeaderSize)
+	}
+	readBuffSize := make([]int, ReadTunBuffNum)
+	ih := PacketHeader{}
+	for {
+		n, err := s.tun.Read(readBuff, readBuffSize, PIHeaderSize)
+		if err != nil {
+			gLog.Printf(LvERROR, "read tun fail: ", err)
+			return
+		}
+		for i := 0; i < n; i++ {
+			if readBuffSize[i] > ReadTunBuffSize {
+				gLog.Printf(LvERROR, "read tun overflow: len=", readBuffSize[i])
+				continue
+			}
+			parseHeader(readBuff[i][PIHeaderSize:readBuffSize[i]+PIHeaderSize], &ih)
+			gLog.Printf(LvDev, "read tun dst ip=%s,len=%d", net.IP{byte(ih.dst >> 24), byte(ih.dst >> 16), byte(ih.dst >> 8), byte(ih.dst)}.String(), readBuffSize[0])
+			s.routeTunPacket(readBuff[i][PIHeaderSize:readBuffSize[i]+PIHeaderSize], &ih)
+		}
+	}
+}
+
+func (s *p2pSDWAN) StartTun() error {
+	sdwan := gConf.getSDWAN()
+	if s.tun == nil {
+		tun := &optun{}
+		err := tun.Start(s.virtualIP.String(), &sdwan)
+		if err != nil {
+			gLog.Println(LvERROR, "open tun fail:", err)
+			return err
+		}
+		s.tun = tun
+		go s.readTunLoop()
+		go s.readNodeLoop() // multi-thread read will cause packets out of order, resulting in slower speeds
+	}
+	err := setTunAddr(s.tun.tunName, s.virtualIP.String(), sdwan.Gateway, s.tun.dev)
+	if err != nil {
+		gLog.Printf(LvERROR, "setTunAddr error:%s,%s,%s,%s", err, s.tun.tunName, s.virtualIP.String(), sdwan.Gateway)
+		return err
+	}
+	return nil
+}
+
+func handleSDWAN(subType uint16, msg []byte) error {
+	gLog.Printf(LvDEBUG, "handle sdwan msg type:%d", subType)
+	var err error
+	switch subType {
+	case MsgSDWANInfoRsp:
+		rsp := SDWANInfo{}
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp); err != nil {
+			return ErrMsgFormat
+		}
+		gLog.Println(LvINFO, "sdwan init:", prettyJson(rsp))
+		if runtime.GOOS == "android" {
+			AndroidSDWANConfig <- msg[openP2PHeaderSize:]
+		}
+		// GNetwork.sdwan.detail = &rsp
+		gConf.setSDWAN(rsp)
+		err = GNetwork.sdwan.init(gConf.Network.Node)
+		if err != nil {
+			gLog.Println(LvERROR, "sdwan init fail: ", err)
+			if GNetwork.sdwan.tun != nil {
+				GNetwork.sdwan.tun.Stop()
+				GNetwork.sdwan.tun = nil
+				return err
+			}
+		}
+		go GNetwork.sdwan.run()
+	default:
+	}
+	return err
+}

core/speedlimiter.go

@@ -0,0 +1,50 @@
+package openp2p
+
+import (
+	"sync"
+	"time"
+)
+
+// SpeedLimiter ...
+type SpeedLimiter struct {
+	lastUpdate time.Time
+	speed      int // per second
+	precision  int // seconds
+	freeCap    int
+	maxFreeCap int
+	mtx        sync.Mutex
+}
+
+func newSpeedLimiter(speed int, precision int) *SpeedLimiter {
+	return &SpeedLimiter{
+		speed:      speed,
+		precision:  precision,
+		lastUpdate: time.Now(),
+		maxFreeCap: speed * precision,
+		freeCap:    speed * precision,
+	}
+}
+
+// Add ...
+func (sl *SpeedLimiter) Add(increment int, wait bool) bool {
+	if sl.speed <= 0 {
+		return true
+	}
+	sl.mtx.Lock()
+	defer sl.mtx.Unlock()
+	sl.freeCap += int(time.Since(sl.lastUpdate) * time.Duration(sl.speed) / time.Second)
+	if sl.freeCap > sl.maxFreeCap {
+		sl.freeCap = sl.maxFreeCap
+	}
+	if !wait && sl.freeCap < increment {
+		return false
+	}
+	sl.freeCap -= increment
+	sl.lastUpdate = time.Now()
+	if sl.freeCap < 0 {
+		// sleep for the overflow
+		// fmt.Println("sleep ", time.Millisecond*time.Duration(-sl.freeCap*100)/time.Duration(sl.speed))
+		time.Sleep(time.Millisecond * time.Duration(-sl.freeCap*1000) / time.Duration(sl.speed)) // sleep ms
+	}
+	return true
+}

core/speedlimiter_test.go

@@ -0,0 +1,59 @@
+package openp2p
+
+import (
+	"testing"
+	"time"
+)
+
+func TestBandwidth(t *testing.T) {
+	speed := 10 * 1024 * 1024 / 8 // 10mbps
+	speedl := newSpeedLimiter(speed, 1)
+	oneBuffSize := 4096
+	writeNum := 5000
+	expectTime := oneBuffSize * writeNum / speed
+	startTs := time.Now()
+	for i := 0; i < writeNum; i++ {
+		speedl.Add(oneBuffSize, true)
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}
+
+func TestSymmetric(t *testing.T) {
+	speed := 20000 / 180
+	speedl := newSpeedLimiter(speed, 180)
+	oneBuffSize := 300
+	writeNum := 70
+	expectTime := (oneBuffSize*writeNum - 20000) / speed
+	t.Logf("expect %ds", expectTime)
+	startTs := time.Now()
+	for i := 0; i < writeNum; i++ {
+		speedl.Add(oneBuffSize, true)
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}
+
+func TestSymmetric2(t *testing.T) {
+	speed := 30000 / 180
+	speedl := newSpeedLimiter(speed, 180)
+	oneBuffSize := 800
+	writeNum := 40
+	expectTime := (oneBuffSize*writeNum - 30000) / speed
+	startTs := time.Now()
+	for i := 0; i < writeNum; {
+		if speedl.Add(oneBuffSize, true) {
+			i++
+		} else {
+			time.Sleep(time.Millisecond)
+		}
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}

core/udp.go

@@ -18,7 +18,7 @@ func UDPWrite(conn *net.UDPConn, dst net.Addr, mainType uint16, subType uint16,
 	return conn.WriteTo(msg, dst)
 }
 
-func UDPRead(conn *net.UDPConn, timeout time.Duration) (ra net.Addr, head *openP2PHeader, result []byte, len int, err error) {
+func UDPRead(conn *net.UDPConn, timeout time.Duration) (ra net.Addr, head *openP2PHeader, buff []byte, length int, err error) {
 	if timeout > 0 {
 		err = conn.SetReadDeadline(time.Now().Add(timeout))
 		if err != nil {
@@ -27,15 +27,15 @@ func UDPRead(conn *net.UDPConn, timeout time.Duration) (ra net.Addr, head *openP
 		}
 	}
 
-	result = make([]byte, 1024)
-	len, ra, err = conn.ReadFrom(result)
+	buff = make([]byte, 1024)
+	length, ra, err = conn.ReadFrom(buff)
 	if err != nil {
 		// gLog.Println(LevelDEBUG, "ReadFrom error")
 		return nil, nil, nil, 0, err
 	}
 	head = &openP2PHeader{}
-	err = binary.Read(bytes.NewReader(result[:openP2PHeaderSize]), binary.LittleEndian, head)
-	if err != nil {
+	err = binary.Read(bytes.NewReader(buff[:openP2PHeaderSize]), binary.LittleEndian, head)
+	if err != nil || head.DataLen > uint32(len(buff)-openP2PHeaderSize) {
 		gLog.Println(LvERROR, "parse p2pheader error:", err)
 		return nil, nil, nil, 0, err
 	}

core/underlay.go

@@ -1,16 +1,65 @@
 package openp2p
 
 import (
+	"io"
 	"time"
 )
 
 type underlay interface {
+	Read([]byte) (int, error)
+	Write([]byte) (int, error)
 	ReadBuffer() (*openP2PHeader, []byte, error)
 	WriteBytes(uint16, uint16, []byte) error
 	WriteBuffer([]byte) error
 	WriteMessage(uint16, uint16, interface{}) error
 	Close() error
+	WLock()
+	WUnlock()
 	SetReadDeadline(t time.Time) error
 	SetWriteDeadline(t time.Time) error
 	Protocol() string
 }
+
+func DefaultReadBuffer(ul underlay) (*openP2PHeader, []byte, error) {
+	headBuf := make([]byte, openP2PHeaderSize)
+	_, err := io.ReadFull(ul, headBuf)
+	if err != nil {
+		return nil, nil, err
+	}
+	head, err := decodeHeader(headBuf)
+	if err != nil || head.MainType > 16 {
+		return nil, nil, err
+	}
+	dataBuf := make([]byte, head.DataLen)
+	_, err = io.ReadFull(ul, dataBuf)
+	return head, dataBuf, err
+}
+
+func DefaultWriteBytes(ul underlay, mainType, subType uint16, data []byte) error {
+	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err := ul.Write(writeBytes)
+	ul.WUnlock()
+	return err
+}
+
+func DefaultWriteBuffer(ul underlay, data []byte) error {
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err := ul.Write(data)
+	ul.WUnlock()
+	return err
+}
+
+func DefaultWriteMessage(ul underlay, mainType uint16, subType uint16, packet interface{}) error {
+	writeBytes, err := newMessage(mainType, subType, packet)
+	if err != nil {
+		return err
+	}
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err = ul.Write(writeBytes)
+	ul.WUnlock()
+	return err
+}

core/underlay_kcp.go

@@ -0,0 +1,94 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"sync"
+	"time"
+
+	"github.com/xtaci/kcp-go/v5"
+)
+
+type underlayKCP struct {
+	listener *kcp.Listener
+	writeMtx *sync.Mutex
+	*kcp.UDPSession
+}
+
+func (conn *underlayKCP) Protocol() string {
+	return "kcp"
+}
+
+func (conn *underlayKCP) ReadBuffer() (*openP2PHeader, []byte, error) {
+	return DefaultReadBuffer(conn)
+}
+
+func (conn *underlayKCP) WriteBytes(mainType uint16, subType uint16, data []byte) error {
+	return DefaultWriteBytes(conn, mainType, subType, data)
+}
+
+func (conn *underlayKCP) WriteBuffer(data []byte) error {
+	return DefaultWriteBuffer(conn, data)
+}
+
+func (conn *underlayKCP) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
+	return DefaultWriteMessage(conn, mainType, subType, packet)
+}
+
+func (conn *underlayKCP) Close() error {
+	conn.UDPSession.Close()
+	return nil
+}
+func (conn *underlayKCP) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayKCP) WUnlock() {
+	conn.writeMtx.Unlock()
+}
+func (conn *underlayKCP) CloseListener() {
+	if conn.listener != nil {
+		conn.listener.Close()
+	}
+}
+
+func (conn *underlayKCP) Accept() error {
+	kConn, err := conn.listener.AcceptKCP()
+	if err != nil {
+		conn.listener.Close()
+		return err
+	}
+	kConn.SetNoDelay(0, 40, 0, 0)
+	kConn.SetWindowSize(512, 512)
+	kConn.SetWriteBuffer(1024 * 128)
+	kConn.SetReadBuffer(1024 * 128)
+	conn.UDPSession = kConn
+	return nil
+}
+
+func listenKCP(addr string, idleTimeout time.Duration) (*underlayKCP, error) {
+	gLog.Println(LvDEBUG, "kcp listen on ", addr)
+	listener, err := kcp.ListenWithOptions(addr, nil, 0, 0)
+	if err != nil {
+		return nil, fmt.Errorf("quic.ListenAddr error:%s", err)
+	}
+	ul := &underlayKCP{listener: listener, writeMtx: &sync.Mutex{}}
+	err = ul.Accept()
+	if err != nil {
+		ul.CloseListener()
+		return nil, fmt.Errorf("accept KCP error:%s", err)
+	}
+	return ul, nil
+}
+
+func dialKCP(conn *net.UDPConn, remoteAddr *net.UDPAddr, idleTimeout time.Duration) (*underlayKCP, error) {
+	kConn, err := kcp.NewConn(remoteAddr.String(), nil, 0, 0, conn)
+	if err != nil {
+		return nil, fmt.Errorf("quic.DialContext error:%s", err)
+	}
+	kConn.SetNoDelay(0, 40, 0, 0)
+	kConn.SetWindowSize(512, 512)
+	kConn.SetWriteBuffer(1024 * 128)
+	kConn.SetReadBuffer(1024 * 128)
+	ul := &underlayKCP{nil, &sync.Mutex{}, kConn}
+	return ul, nil
+}

core/underlay_quic.go

@@ -6,10 +6,8 @@ import (
 	"crypto/rsa"
 	"crypto/tls"
 	"crypto/x509"
-	"encoding/json"
 	"encoding/pem"
 	"fmt"
-	"io"
 	"math/big"
 	"net"
 	"sync"
@@ -33,53 +31,33 @@ func (conn *underlayQUIC) Protocol() string {
 }
 
 func (conn *underlayQUIC) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
+	return DefaultReadBuffer(conn)
 }
 
 func (conn *underlayQUIC) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBytes(conn, mainType, subType, data)
 }
 
 func (conn *underlayQUIC) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBuffer(conn, data)
 }
 
 func (conn *underlayQUIC) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteMessage(conn, mainType, subType, packet)
 }
 
 func (conn *underlayQUIC) Close() error {
 	conn.Stream.CancelRead(1)
 	conn.Connection.CloseWithError(0, "")
+	conn.CloseListener()
 	return nil
 }
+func (conn *underlayQUIC) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayQUIC) WUnlock() {
+	conn.writeMtx.Unlock()
+}
 func (conn *underlayQUIC) CloseListener() {
 	if conn.listener != nil {
 		conn.listener.Close()
@@ -109,7 +87,13 @@ func listenQuic(addr string, idleTimeout time.Duration) (*underlayQUIC, error) {
 	if err != nil {
 		return nil, fmt.Errorf("quic.ListenAddr error:%s", err)
 	}
-	return &underlayQUIC{listener: listener, writeMtx: &sync.Mutex{}}, nil
+	ul := &underlayQUIC{listener: listener, writeMtx: &sync.Mutex{}}
+	err = ul.Accept()
+	if err != nil {
+		ul.CloseListener()
+		return nil, fmt.Errorf("accept quic error:%s", err)
+	}
+	return ul, nil
 }
 
 func dialQuic(conn *net.UDPConn, remoteAddr *net.UDPAddr, idleTimeout time.Duration) (*underlayQUIC, error) {

core/underlay_tcp.go

@@ -1,9 +1,8 @@
 package openp2p
 
 import (
-	"encoding/json"
+	"encoding/binary"
 	"fmt"
-	"io"
 	"net"
 	"sync"
 	"time"
@@ -14,6 +13,7 @@ import (
 type underlayTCP struct {
 	writeMtx *sync.Mutex
 	net.Conn
+	connectTime time.Time
 }
 
 func (conn *underlayTCP) Protocol() string {
@@ -21,97 +21,115 @@ func (conn *underlayTCP) Protocol() string {
 }
 
 func (conn *underlayTCP) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
+	return DefaultReadBuffer(conn)
 }
 
 func (conn *underlayTCP) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBytes(conn, mainType, subType, data)
 }
 
 func (conn *underlayTCP) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBuffer(conn, data)
 }
 
 func (conn *underlayTCP) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteMessage(conn, mainType, subType, packet)
 }
 
 func (conn *underlayTCP) Close() error {
 	return conn.Conn.Close()
 }
+func (conn *underlayTCP) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayTCP) WUnlock() {
+	conn.writeMtx.Unlock()
+}
 
 func listenTCP(host string, port int, localPort int, mode string, t *P2PTunnel) (*underlayTCP, error) {
 	if mode == LinkModeTCPPunch {
-		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) == LESS {
+		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
 			gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
 		} else {
-			ts := time.Duration(int64(t.punchTs) + t.pn.dt - time.Now().UnixNano())
+			ts := time.Duration(int64(t.punchTs) + GNetwork.dt - time.Now().UnixNano())
 			gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
 			time.Sleep(ts)
 		}
-		gLog.Println(LvDEBUG, (time.Now().UnixNano()-t.pn.dt)/(int64)(time.Millisecond), " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", localPort), "-->", fmt.Sprintf("%s:%d", host, port))
+		gLog.Println(LvDEBUG, " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", localPort), "-->", fmt.Sprintf("%s:%d", host, port))
 		c, err := reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout)
 		if err != nil {
 			gLog.Println(LvDEBUG, "send tcp punch: ", err)
 			return nil, err
 		}
-		return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
-	}
-	t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-	addr, _ := net.ResolveTCPAddr("tcp", fmt.Sprintf("0.0.0.0:%d", localPort))
-	l, err := net.ListenTCP("tcp", addr)
+		utcp := &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}
+		_, buff, err := utcp.ReadBuffer()
 		if err != nil {
+			return nil, fmt.Errorf("read start msg error:%s", err)
+		}
+		if buff != nil {
+			gLog.Println(LvDEBUG, string(buff))
+		}
+		utcp.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, buff)
+		return utcp, nil
+	}
+	GNetwork.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+	tid := t.id
+	if compareVersion(t.config.peerVersion, PublicIPVersion) < 0 { // old version
+		ipBytes := net.ParseIP(t.config.peerIP).To4()
+		tid = uint64(binary.BigEndian.Uint32(ipBytes))
+		gLog.Println(LvDEBUG, "compatible with old client, use ip as key:", tid)
+	}
+	var utcp *underlayTCP
+	if mode == LinkModeIntranet && gConf.Network.hasIPv4 == 0 && gConf.Network.hasUPNPorNATPMP == 0 {
+		addr, _ := net.ResolveTCPAddr("tcp4", fmt.Sprintf("0.0.0.0:%d", localPort))
+		l, err := net.ListenTCP("tcp4", addr)
+		if err != nil {
+			gLog.Printf(LvERROR, "listen %d error:", localPort, err)
 			return nil, err
 		}
-	l.SetDeadline(time.Now().Add(CheckActiveTimeout))
-	c, err := l.Accept()
 		defer l.Close()
+		err = l.SetDeadline(time.Now().Add(UnderlayTCPConnectTimeout))
+		if err != nil {
+			gLog.Printf(LvERROR, "set listen timeout:", err)
+			return nil, err
+		}
+		c, err := l.Accept()
 		if err != nil {
 			return nil, err
 		}
-	return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
+		utcp = &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}
+	} else {
+		if v4l != nil {
+			utcp = v4l.getUnderlayTCP(tid)
+		}
+	}
+
+	if utcp == nil {
+		return nil, ErrConnectPublicV4
+	}
+	return utcp, nil
 }
 
 func dialTCP(host string, port int, localPort int, mode string) (*underlayTCP, error) {
 	var c net.Conn
 	var err error
 	if mode == LinkModeTCPPunch {
-		c, err = reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout)
+		gLog.Println(LvDev, " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", localPort), "-->", fmt.Sprintf("%s:%d", host, port))
+		if c, err = reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout); err != nil {
+			gLog.Println(LvDev, "send tcp punch: ", err)
+		}
+
 	} else {
 		c, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout)
 	}
 
 	if err != nil {
-		gLog.Printf(LvERROR, "Dial %s:%d error:%s", host, port, err)
+		gLog.Printf(LvDev, "Dial %s:%d error:%s", host, port, err)
 		return nil, err
 	}
+	tc := c.(*net.TCPConn)
+	tc.SetKeepAlive(true)
+	tc.SetKeepAlivePeriod(UnderlayTCPKeepalive)
 	gLog.Printf(LvDEBUG, "Dial %s:%d OK", host, port)
 	return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
 }

core/underlay_tcp6.go

@@ -1,16 +1,13 @@
 package openp2p
 
 import (
-	"encoding/json"
 	"fmt"
-	"io"
 	"net"
 	"sync"
 	"time"
 )
 
 type underlayTCP6 struct {
-	listener net.Listener
 	writeMtx *sync.Mutex
 	net.Conn
 }
@@ -20,60 +17,38 @@ func (conn *underlayTCP6) Protocol() string {
 }
 
 func (conn *underlayTCP6) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
+	return DefaultReadBuffer(conn)
 }
 
 func (conn *underlayTCP6) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBytes(conn, mainType, subType, data)
 }
 
 func (conn *underlayTCP6) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteBuffer(conn, data)
 }
 
 func (conn *underlayTCP6) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
+	return DefaultWriteMessage(conn, mainType, subType, packet)
 }
 
 func (conn *underlayTCP6) Close() error {
 	return conn.Conn.Close()
 }
-
-func listenTCP6(port int, idleTimeout time.Duration) (*underlayTCP6, error) {
+func (conn *underlayTCP6) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayTCP6) WUnlock() {
+	conn.writeMtx.Unlock()
+}
+func listenTCP6(port int, timeout time.Duration) (*underlayTCP6, error) {
 	addr, _ := net.ResolveTCPAddr("tcp6", fmt.Sprintf("[::]:%d", port))
 	l, err := net.ListenTCP("tcp6", addr)
 	if err != nil {
 		return nil, err
 	}
 	defer l.Close()
-	l.SetDeadline(time.Now().Add(HandshakeTimeout))
+	l.SetDeadline(time.Now().Add(timeout))
 	c, err := l.Accept()
 	defer l.Close()
 	if err != nil {
@@ -83,7 +58,7 @@ func listenTCP6(port int, idleTimeout time.Duration) (*underlayTCP6, error) {
 }
 
 func dialTCP6(host string, port int) (*underlayTCP6, error) {
-	c, err := net.DialTimeout("tcp6", fmt.Sprintf("[%s]:%d", host, port), HandshakeTimeout)
+	c, err := net.DialTimeout("tcp6", fmt.Sprintf("[%s]:%d", host, port), UnderlayConnectTimeout)
 	if err != nil {
 		gLog.Printf(LvERROR, "Dial %s:%d error:%s", host, port, err)
 		return nil, err

core/update.go

@@ -11,6 +11,7 @@ import (
 	"io"
 	"io/ioutil"
 	"net/http"
+	"net/url"
 	"os"
 	"path/filepath"
 	"runtime"
@@ -38,7 +39,7 @@ func update(host string, port int) error {
 	}
 	goos := runtime.GOOS
 	goarch := runtime.GOARCH
-	rsp, err := c.Get(fmt.Sprintf("https://%s:%d/api/v1/update?fromver=%s&os=%s&arch=%s&user=%s&node=%s", host, port, OpenP2PVersion, goos, goarch, gConf.Network.User, gConf.Network.Node))
+	rsp, err := c.Get(fmt.Sprintf("https://%s:%d/api/v1/update?fromver=%s&os=%s&arch=%s&user=%s&node=%s", host, port, OpenP2PVersion, goos, goarch, url.QueryEscape(gConf.Network.User), url.QueryEscape(gConf.Network.Node)))
 	if err != nil {
 		gLog.Println(LvERROR, "update:query update list failed:", err)
 		return err
@@ -70,13 +71,10 @@ func update(host string, port int) error {
 	return nil
 }
 
-// todo rollback on error
-func updateFile(url string, checksum string, dst string) error {
-	gLog.Println(LvINFO, "download ", url)
-	tmpFile := filepath.Dir(os.Args[0]) + "/openp2p.tmp"
-	output, err := os.OpenFile(tmpFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0776)
+func downloadFile(url string, checksum string, dstFile string) error {
+	output, err := os.OpenFile(dstFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0776)
 	if err != nil {
-		gLog.Printf(LvERROR, "OpenFile %s error:%s", tmpFile, err)
+		gLog.Printf(LvERROR, "OpenFile %s error:%s", dstFile, err)
 		return err
 	}
 	caCertPool, err := x509.SystemCertPool()
@@ -110,11 +108,22 @@ func updateFile(url string, checksum string, dst string) error {
 	output.Close()
 	gLog.Println(LvINFO, "download ", url, " ok")
 	gLog.Printf(LvINFO, "size: %d bytes", n)
+	return nil
+}
 
-	err = os.Rename(os.Args[0], os.Args[0]+"0") // the old daemon process was using the 0 file, so it will prevent override it
+func updateFile(url string, checksum string, dst string) error {
+	gLog.Println(LvINFO, "download ", url)
+	tmpFile := filepath.Dir(os.Args[0]) + "/openp2p.tmp"
+	err := downloadFile(url, checksum, tmpFile)
+	if err != nil {
+		return err
+	}
+	backupFile := os.Args[0] + "0"
+	err = os.Rename(os.Args[0], backupFile) // the old daemon process was using the 0 file, so it will prevent override it
 	if err != nil {
 		gLog.Printf(LvINFO, " rename %s error:%s, retry 1", os.Args[0], err)
-		err = os.Rename(os.Args[0], os.Args[0]+"1")
+		backupFile = os.Args[0] + "1"
+		err = os.Rename(os.Args[0], backupFile)
 		if err != nil {
 			gLog.Printf(LvINFO, " rename %s error:%s", os.Args[0], err)
 		}
@@ -124,7 +133,7 @@ func updateFile(url string, checksum string, dst string) error {
 	err = extract(filepath.Dir(os.Args[0]), tmpFile)
 	if err != nil {
 		gLog.Printf(LvERROR, "extract error:%s. revert rename", err)
-		os.Rename(os.Args[0]+"0", os.Args[0])
+		os.Rename(backupFile, os.Args[0])
 		return err
 	}
 	os.Remove(tmpFile)
@@ -203,9 +212,6 @@ func extractTgz(dst, src string) error {
 			if err != nil {
 				return err
 			}
-			if err != nil {
-				return err
-			}
 			defer outFile.Close()
 			if _, err := io.Copy(outFile, tarReader); err != nil {
 				return err
@@ -218,12 +224,16 @@ func extractTgz(dst, src string) error {
 }
 
 func cleanTempFiles() {
-	err := os.Remove(os.Args[0] + "0")
-	if err != nil {
-		gLog.Printf(LvDEBUG, " remove %s error:%s", os.Args[0]+"0", err)
+	tmpFile := os.Args[0] + "0"
+	if _, err := os.Stat(tmpFile); err == nil {
+		if err := os.Remove(tmpFile); err != nil {
+			gLog.Printf(LvDEBUG, " remove %s error:%s", tmpFile, err)
+		}
+	}
+	tmpFile = os.Args[0] + "1"
+	if _, err := os.Stat(tmpFile); err == nil {
+		if err := os.Remove(tmpFile); err != nil {
+			gLog.Printf(LvDEBUG, " remove %s error:%s", tmpFile, err)
 		}
-	err = os.Remove(os.Args[0] + "1")
-	if err != nil {
-		gLog.Printf(LvDEBUG, " remove %s error:%s", os.Args[0]+"0", err)
 	}
 }

core/upnp.go

@@ -358,11 +358,6 @@ func (n *upnpNAT) AddPortMapping(protocol string, externalPort, internalPort int
 		return
 	}
 
-	// TODO: check response to see if the port was forwarded
-	// log.Println(message, response)
-	// JAE:
-	// body, err := ioutil.ReadAll(response.Body)
-	// fmt.Println(string(body), err)
 	mappedExternalPort = externalPort
 	_ = response
 	return
@@ -384,8 +379,6 @@ func (n *upnpNAT) DeletePortMapping(protocol string, externalPort, internalPort
 		return
 	}
 
-	// TODO: check response to see if the port was deleted
-	// log.Println(message, response)
 	_ = response
 	return
 }

core/util_windows.go

@@ -5,6 +5,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"strconv"
 	"strings"
 
 	"golang.org/x/sys/windows/registry"
@@ -23,6 +24,17 @@ func getOsName() (osName string) {
 	defer k.Close()
 	pn, _, err := k.GetStringValue("ProductName")
 	if err == nil {
+		currentBuild, _, err := k.GetStringValue("CurrentBuild")
+		if err != nil {
+			return
+		}
+		buildNumber, err := strconv.Atoi(currentBuild)
+		if err != nil {
+			return
+		}
+		if buildNumber >= 22000 {
+			pn = strings.Replace(pn, "Windows 10", "Windows 11", 1)
+		}
 		osName = pn
 	}
 	return

core/v4listener.go

@@ -0,0 +1,92 @@
+package openp2p
+
+import (
+	"encoding/binary"
+	"fmt"
+	"net"
+	"sync"
+	"time"
+)
+
+type v4Listener struct {
+	conns    sync.Map
+	port     int
+	acceptCh chan bool
+}
+
+func (vl *v4Listener) start() error {
+	v4l.acceptCh = make(chan bool, 500)
+	for {
+		vl.listen()
+		time.Sleep(UnderlayTCPConnectTimeout)
+	}
+}
+
+func (vl *v4Listener) listen() error {
+	gLog.Printf(LvINFO, "v4Listener listen %d start", vl.port)
+	defer gLog.Printf(LvINFO, "v4Listener listen %d end", vl.port)
+	addr, _ := net.ResolveTCPAddr("tcp4", fmt.Sprintf("0.0.0.0:%d", vl.port))
+	l, err := net.ListenTCP("tcp4", addr)
+	if err != nil {
+		gLog.Printf(LvERROR, "v4Listener listen %d error:", vl.port, err)
+		return err
+	}
+	defer l.Close()
+	for {
+		c, err := l.Accept()
+		if err != nil {
+			break
+		}
+		go vl.handleConnection(c)
+	}
+	return nil
+}
+func (vl *v4Listener) handleConnection(c net.Conn) {
+	gLog.Println(LvDEBUG, "v4Listener accept connection: ", c.RemoteAddr().String())
+	utcp := &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c, connectTime: time.Now()}
+	utcp.SetReadDeadline(time.Now().Add(UnderlayTCPConnectTimeout))
+	_, buff, err := utcp.ReadBuffer()
+	if err != nil {
+		gLog.Println(LvERROR, "utcp.ReadBuffer error:", err)
+	}
+	utcp.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, buff)
+	var tid uint64
+	if string(buff) == "OpenP2P,hello" { // old client
+		// save remoteIP as key
+		remoteAddr := c.RemoteAddr().(*net.TCPAddr).IP
+		ipBytes := remoteAddr.To4()
+		tid = uint64(binary.BigEndian.Uint32(ipBytes)) // bytes not enough for uint64
+		gLog.Println(LvDEBUG, "hello ", string(buff))
+	} else {
+		if len(buff) < 8 {
+			return
+		}
+		tid = binary.LittleEndian.Uint64(buff[:8])
+		gLog.Println(LvDEBUG, "hello ", tid)
+	}
+	// clear timeout connection
+	vl.conns.Range(func(idx, i interface{}) bool {
+		ut := i.(*underlayTCP)
+		if ut.connectTime.Before(time.Now().Add(-UnderlayTCPConnectTimeout)) {
+			vl.conns.Delete(idx)
+		}
+		return true
+	})
+	vl.conns.Store(tid, utcp)
+	if len(vl.acceptCh) == 0 {
+		vl.acceptCh <- true
+	}
+}
+
+func (vl *v4Listener) getUnderlayTCP(tid uint64) *underlayTCP {
+	for i := 0; i < 100; i++ {
+		select {
+		case <-time.After(time.Millisecond * 50):
+		case <-vl.acceptCh:
+		}
+		if u, ok := vl.conns.LoadAndDelete(tid); ok {
+			return u.(*underlayTCP)
+		}
+	}
+	return nil
+}

docker/Dockerfile

@@ -2,7 +2,7 @@ FROM  alpine:3.18.2
 
 # Replace the default Alpine repositories with Aliyun mirrors
 RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && \
-    apk add --no-cache ca-certificates && \
+    apk add --no-cache ca-certificates iptables && \
     rm -rf /tmp/* /var/tmp/* /var/cache/apk/* /var/cache/distfiles/*
 
 COPY get-client.sh /

example/dll/dll.cpp

@@ -0,0 +1,14 @@
+#include <iostream>
+#include <windows.h>
+
+using namespace std;
+typedef void (*pRun)(const char *);
+
+int main(int argc, char *argv[])
+{
+	HMODULE dll = LoadLibraryA("openp2p.dll");
+	pRun run = (pRun)GetProcAddress(dll, "RunCmd");
+	run("-node 5800-debug2 -token YOUR-TOKEN");
+	FreeLibrary(dll);
+	return 0;
+}

example/echo/echo-client.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"fmt"
+	op "openp2p/core"
+	"time"
+)
+
+func main() {
+	op.Run()
+	for i := 0; i < 10; i++ {
+		go echoClient("5800-debug")
+	}
+	echoClient("5800-debug")
+}
+
+func echoClient(peerNode string) {
+	sendDatalen := op.ReadBuffLen
+	sendBuff := make([]byte, sendDatalen)
+	for i := 0; i < len(sendBuff); i++ {
+		sendBuff[i] = byte('A' + i/100)
+	}
+	// peerNode = "YOUR-PEER-NODE-NAME"
+	if err := op.GNetwork.ConnectNode(peerNode); err != nil {
+		fmt.Println("connect error:", err)
+		return
+	}
+	for i := 0; ; i++ {
+		sendBuff[1] = 'A' + byte(i%26)
+		if err := op.GNetwork.WriteNode(op.NodeNameToID(peerNode), sendBuff[:sendDatalen]); err != nil {
+			fmt.Println("write error:", err)
+			break
+		}
+		nd := op.GNetwork.ReadNode(time.Second * 10)
+		if nd == nil {
+			fmt.Printf("waiting for node data\n")
+			time.Sleep(time.Second * 10)
+			continue
+		}
+		fmt.Printf("read %d len=%d data=%s\n", nd.NodeID, len(nd.Data), nd.Data[:16]) // only print 16 bytes
+	}
+}

example/echo/echo-server.go

@@ -0,0 +1,32 @@
+package main
+
+import (
+	"fmt"
+	op "openp2p/core"
+	"time"
+)
+
+func main() {
+	op.Run()
+	echoServer()
+	forever := make(chan bool)
+	<-forever
+}
+
+func echoServer() {
+	// peerID := fmt.Sprintf("%d", core.NodeNameToID(peerNode))
+	for {
+		nd := op.GNetwork.ReadNode(time.Second * 10)
+		if nd == nil {
+			fmt.Printf("waiting for node data\n")
+			// time.Sleep(time.Second * 10)
+			continue
+		}
+		// fmt.Printf("read %s len=%d data=%s\n", nd.Node, len(nd.Data), nd.Data[:16])
+		nd.Data[0] = 'R' // echo server mark as replied
+		if err := op.GNetwork.WriteNode(nd.NodeID, nd.Data); err != nil {
+			fmt.Println("write error:", err)
+			break
+		}
+	}
+}

go.mod

@@ -1,29 +1,41 @@
 module openp2p
 
-go 1.18
+go 1.20
 
 require (
 	github.com/emirpasic/gods v1.18.1
 	github.com/gorilla/websocket v1.4.2
 	github.com/openp2p-cn/go-reuseport v0.3.2
 	github.com/openp2p-cn/service v1.0.0
-	github.com/openp2p-cn/totp v0.0.0-20230102121327-8e02f6b392ed
+	github.com/openp2p-cn/totp v0.0.0-20230421034602-0f3320ffb25e
 	github.com/quic-go/quic-go v0.34.0
-	golang.org/x/sys v0.5.0
+	github.com/vishvananda/netlink v1.1.1-0.20211118161826-650dca95af54
+	github.com/xtaci/kcp-go/v5 v5.5.17
+	golang.org/x/sys v0.26.0
+	golang.zx2c4.com/wireguard/windows v0.5.3
 )
 
 require (
 	github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
-	github.com/golang/mock v1.6.0 // indirect
+	github.com/golang/mock v1.7.0-rc.1 // indirect
 	github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect
 	github.com/kardianos/service v1.2.2 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.5 // indirect
+	github.com/klauspost/reedsolomon v1.11.8 // indirect
 	github.com/onsi/ginkgo/v2 v2.2.0 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
 	github.com/quic-go/qtls-go1-19 v0.3.2 // indirect
 	github.com/quic-go/qtls-go1-20 v0.2.2 // indirect
-	golang.org/x/crypto v0.4.0 // indirect
-	golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect
-	golang.org/x/mod v0.6.0 // indirect
-	golang.org/x/net v0.7.0 // indirect
-	golang.org/x/tools v0.2.0 // indirect
-	google.golang.org/protobuf v1.28.1 // indirect
+	github.com/templexxx/cpu v0.1.0 // indirect
+	github.com/templexxx/xorsimd v0.4.2 // indirect
+	github.com/tjfoc/gmsm v1.4.1 // indirect
+	github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f // indirect
+	golang.org/x/crypto v0.28.0 // indirect
+	golang.org/x/exp v0.0.0-20230725093048-515e97ebf090 // indirect
+	golang.org/x/mod v0.21.0 // indirect
+	golang.org/x/net v0.30.0 // indirect
+	golang.org/x/tools v0.26.0 // indirect
+	golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
+	google.golang.org/protobuf v1.33.0 // indirect
+	gvisor.dev/gvisor v0.0.0-20241128011400-745828301c93 // indirect
 )

lib/openp2p.go

@@ -0,0 +1,18 @@
+package main
+
+// On Windows env
+// cd lib
+// go build -o openp2p.dll -buildmode=c-shared openp2p.go
+// caller example see example/dll
+import (
+	op "openp2p/core"
+)
+import "C"
+
+func main() {
+}
+
+//export RunCmd
+func RunCmd(cmd *C.char) {
+	op.RunCmd(C.GoString(cmd))
+}