WIP: adding binary transfer with 3 MB/s performance

doc/UtpBinaryTransfer.md

@@ -0,0 +1,65 @@
+# UTP in IPv8
+
+uTP is a protocol based on the UDP used to share files in a peer-to-peer fashion. In this project, we have integrated an existing `utp4j` library with the IPv8 infrastructure to allow for binary data transfer to users on the existing network.
+
+## UtpCommunity
+
+To distinguish between the capabilities of different peers, we have created our own `UtpCommunity` to show that the peer supports uTP transfer. We use this community to implement some useful control messages in our infrastructure. These are:
+- `UTP_HEARTBEAT` - used to signal that the peer has a working uTP endpoint and can accept connections.
+- `UTP_TRANSFER_REQUEST` - used as a handshake for establishing a file transfer.
+
+### UtpHelper
+
+To separate some of the logic of sending messages over the protocol, we have created a `UtpHelper` class. It holds all the utility methods used by peers sending data and creates a coroutine to send heartbeat messages.
+
+#### Heartbeats
+
+The heartbeat service is started right after `UtpCommunity` initialization. It creates a separate coroutine which is active as long as the uTP endpoint is open. A heartbeat message is sent every 5 seconds to each peer in the community to signal own liveness.
+
+For now the heartbeat has an empty payload, but could be used to hold some useful data to be sent to all the peers.
+
+#### Data transfer
+
+For file transfer we have two separate methods exposed to the other classes. One allows to send file data directly, the other is used to generate and send a random, hash verified data. The transfer works as follows:
+- First the `UTP_TRANSFER_REQUEST` message is sent to request to the other peer that we want to send them a data with certain metadata (name, size, type (random/file)).
+- We wait for the response from the other peer. They can either accept or decline.
+- If they decline, we abort the transfer. If it is accepted, we send the data using the uTP endpoint.
+
+Peers which have not established the handshake cannot transfer data. All the packets that would be sent directly to the endpoint, would be dropped. This mechanism is used as a basic protection against the DOS attacks. Current defaults always accept the connection, but further conditions could be added to combat malicious transfers.  
+
+## UtpIPv8Endpoint
+
+This endpoint is the main element of the uTP transfer. It uses the previously mentioned existing library to establish transfer of data between two peers using UDP. It is an extension of the existing `UdpEndpoint`. The existing endpoint creates an instance of `UtpIPv8Endpoint` similar to what TFTP has done. Then every packet that is intended to be sent to our endpoint is prefixed with a special byte (`0x42`) and routed to be further processed.
+
+In the actual endpoint code, we have a logic for both sending (client) and receiving (server) data. Most of the code there is based on the examples shown in the library already, just refactored to use Kotlin features. 
+
+Every routed packed is processed in the `onPacket` function. Stripped off the prefix, checked for the handshake details and finally forwarded to the custom client and server socket. 
+
+### UtpSocket
+
+As we are using IPv8 layer for our transfer, we could not use usual implementations of the `DatagramSocket`. We have created a middle layer in-between the IPv8 and `utp4j` library to allow for a seamless integration. It uses the raw IPv8 socket to send messages prefixed with out special byte. For receiving it has a separate receiving channel which is filled by the `onPacket` function in the endpoint. This gives us an elegant solution, which does not interfere with the code of the library.
+
+## Testing
+
+All the code has been tested and achieved 86% instruction coverage.
+
+## Remaining issues
+
+Unfortunately due to the timeframe we were not able to fully check the quality of every element and some solutions could be done better. Smaller existing issues are mentioned directly in the code.
+
+The main improvements would be:
+- code structure
+- missing listener API (which would allow for displaying transfer data progress)
+- more robust handshake architecture
+- test code seems to be flaky due to OutOfMemory errors when run in bulk
+
+# See also
+
+
+| Name | Link |
+| :-: | :-: |
+|  Main project issue   |  https://github.com/Tribler/tribler/issues/7911  |
+| Main app description | https://github.com/Eragoneq/trustchain-superapp/blob/master/doc/BlockchainNetworkingProject/UtpTesting.md |
+| Benchmarking done over 10 weeks | https://github.com/Eragoneq/trustchain-superapp/blob/master/doc/BlockchainNetworkingProject/Benchmarking.md |
+| utp4j changes and benchmarks | https://github.com/PieterCarton/utp4j/blob/master/CHANGES.md |
+| kotlin-ipv8 changes | https://github.com/Eragoneq/kotlin-ipv8/blob/master/doc/UtpBinaryTransfer.md |

ipv8-android/src/main/java/nl/tudelft/ipv8/android/IPv8Android.kt

@@ -109,6 +109,7 @@ object IPv8Android {
                     connectivityManager,
                 )
 
+
             val bluetoothManager =
                 application.getSystemService<BluetoothManager>()
                     ?: throw IllegalStateException("BluetoothManager not found")

ipv8/build.gradle

@@ -35,6 +35,11 @@ ktlint {
     }
 }
 
+test {
+    minHeapSize = "512M"
+    maxHeapSize = "2048M"
+}
+
 jacocoTestReport {
     reports {
         xml.required = true
@@ -81,6 +86,7 @@ dependencies {
     // https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on
     implementation group: 'org.bouncycastle', name: 'bcprov-jdk15to18', version: '1.63'
 
+    api 'com.github.PieterCarton:utp4j:v0.0.2'
 
 }
 

ipv8/src/main/java/nl/tudelft/ipv8/messaging/EndpointAggregator.kt

@@ -12,7 +12,7 @@ import java.util.*
  */
 class EndpointAggregator(
     val udpEndpoint: UdpEndpoint?,
-    val bluetoothEndpoint: BluetoothEndpoint?
+    val bluetoothEndpoint: BluetoothEndpoint?,
 ) : Endpoint<Peer>(), EndpointListener {
     private var isOpen: Boolean = false
 

ipv8/src/main/java/nl/tudelft/ipv8/messaging/Serialization.kt

@@ -2,11 +2,15 @@ package nl.tudelft.ipv8.messaging
 
 import java.nio.Buffer
 import java.nio.ByteBuffer
+import kotlin.reflect.full.isSubclassOf
+import kotlin.reflect.full.memberProperties
+import kotlin.reflect.full.primaryConstructor
 
 const val SERIALIZED_USHORT_SIZE = 2
 const val SERIALIZED_UINT_SIZE = 4
+const val SERIALIZED_INT_SIZE = 4
 const val SERIALIZED_ULONG_SIZE = 8
-const val SERIALIZED_LONG_SIZE = 4
+const val SERIALIZED_LONG_SIZE = 8
 const val SERIALIZED_UBYTE_SIZE = 1
 
 const val SERIALIZED_PUBLIC_KEY_SIZE = 74
@@ -20,6 +24,80 @@ interface Serializable {
 
 interface Deserializable<T> {
     fun deserialize(buffer: ByteArray, offset: Int = 0): Pair<T, Int>
+
+}
+
+/**
+ * Serializes the object and returns the buffer.
+ * Alternative to manually defining the serialize function.
+ */
+interface AutoSerializable : Serializable {
+    override fun serialize(): ByteArray {
+        return this::class.primaryConstructor!!.parameters.map { param ->
+            val value =
+                this.javaClass.kotlin.memberProperties.find { it.name == param.name }!!.get(this)
+            simpleSerialize(value)
+        }.reduce { acc, bytes -> acc + bytes }
+    }
+}
+
+///**
+// * Deserializes the object from the buffer and returns the object and the new offset.
+// * Alternative to manually defining the deserialize function.
+// */
+//inline fun <reified T> Deserializable<T>.autoDeserialize(buffer: ByteArray, offset: Int = 0): Pair<T, Int> {
+//    TODO()
+//}
+
+fun <U> simpleSerialize(data: U): ByteArray {
+    return when (data) {
+        is Int -> serializeInt(data)
+        is Long -> serializeLong(data)
+        is UByte -> serializeUChar(data)
+        is UInt -> serializeUInt(data)
+        is UShort -> serializeUShort(data.toInt())
+        is ULong -> serializeULong(data)
+        is String -> serializeVarLen(data.toByteArray())
+        is ByteArray -> serializeVarLen(data)
+        is Boolean -> serializeBool(data)
+        is Enum<*> -> serializeUInt(data.ordinal.toUInt())
+        is Serializable -> data.serialize()
+        else -> throw IllegalArgumentException("Unsupported serialization type")
+    }
+}
+
+inline fun <reified U> simpleDeserialize(buffer: ByteArray, offset: Int = 0): Pair<U, Int> {
+    val (value, off) = when (U::class) {
+        Int::class -> Pair(deserializeInt(buffer, offset) as U, SERIALIZED_INT_SIZE)
+        Long::class -> Pair(deserializeLong(buffer, offset) as U, SERIALIZED_LONG_SIZE)
+        UByte::class -> Pair(deserializeUChar(buffer, offset) as U, SERIALIZED_UBYTE_SIZE)
+        UShort::class -> Pair(
+            deserializeUShort(buffer, offset).toUShort() as U,
+            SERIALIZED_USHORT_SIZE
+        )
+
+        UInt::class -> Pair(deserializeUInt(buffer, offset) as U, SERIALIZED_UINT_SIZE)
+        ULong::class -> Pair(deserializeULong(buffer, offset) as U, SERIALIZED_ULONG_SIZE)
+        String::class -> {
+            val (data, len) = deserializeVarLen(buffer, offset)
+            Pair(data.decodeToString() as U, len)
+        }
+
+        ByteArray::class -> {
+            val (data, len) = deserializeVarLen(buffer, offset)
+            Pair(data as U, len)
+        }
+
+        Boolean::class -> Pair(deserializeBool(buffer, offset) as U, 1)
+        else -> {
+            if (U::class.isSubclassOf(Enum::class)) {
+                val ordinal = deserializeUInt(buffer, offset).toInt()
+                val values = U::class.java.enumConstants
+                Pair(values[ordinal] as U, SERIALIZED_UINT_SIZE)
+            } else throw IllegalArgumentException("Unsupported deserialization type")
+        }
+    }
+    return (value to (offset + off))
 }
 
 fun serializeBool(data: Boolean): ByteArray {
@@ -40,10 +118,21 @@ fun serializeUShort(value: Int): ByteArray {
     return bytes
 }
 
+fun serializeUShort(value: UShort): ByteArray {
+    val bytes = ByteBuffer.allocate(SERIALIZED_USHORT_SIZE)
+    bytes.putShort(value.toShort())
+    return bytes.array()
+}
+
 fun deserializeUShort(buffer: ByteArray, offset: Int = 0): Int {
     return (((buffer[offset].toInt() and 0xFF) shl 8) or (buffer[offset + 1].toInt() and 0xFF))
 }
 
+fun deserializeRealUShort(buffer: ByteArray, offset: Int = 0): UShort {
+    val buf = ByteBuffer.wrap(buffer, offset, SERIALIZED_USHORT_SIZE)
+    return buf.short.toUShort()
+}
+
 fun serializeUInt(value: UInt): ByteArray {
     val bytes = UByteArray(SERIALIZED_UINT_SIZE)
     for (i in 0 until SERIALIZED_UINT_SIZE) {
@@ -80,7 +169,7 @@ fun deserializeULong(buffer: ByteArray, offset: Int = 0): ULong {
 
 fun serializeLong(value: Long): ByteArray {
     val buffer = ByteBuffer.allocate(SERIALIZED_LONG_SIZE)
-    buffer.putInt(value.toInt())
+    buffer.putLong(value)
     return buffer.array()
 }
 
@@ -89,7 +178,20 @@ fun deserializeLong(bytes: ByteArray, offset: Int = 0): Long {
     buffer.put(bytes.copyOfRange(offset, offset + SERIALIZED_LONG_SIZE))
     // In JDK 8 this returns a Buffer.
     (buffer as Buffer).flip()
-    return buffer.int.toLong()
+    return buffer.long
+}
+
+fun serializeInt(value: Int): ByteArray {
+    val buffer = ByteBuffer.allocate(SERIALIZED_INT_SIZE)
+    buffer.putInt(value)
+    return buffer.array()
+}
+
+fun deserializeInt(bytes: ByteArray, offset: Int = 0): Int {
+    val buffer = ByteBuffer.allocate(SERIALIZED_INT_SIZE)
+    buffer.put(bytes.copyOfRange(offset, offset + SERIALIZED_INT_SIZE))
+    buffer.flip()
+    return buffer.int
 }
 
 fun serializeUChar(char: UByte): ByteArray {
@@ -107,8 +209,10 @@ fun serializeVarLen(bytes: ByteArray): ByteArray {
 
 fun deserializeVarLen(buffer: ByteArray, offset: Int = 0): Pair<ByteArray, Int> {
     val len = deserializeUInt(buffer, offset).toInt()
-    val payload = buffer.copyOfRange(offset + SERIALIZED_UINT_SIZE,
-        offset + SERIALIZED_UINT_SIZE + len)
+    val payload = buffer.copyOfRange(
+        offset + SERIALIZED_UINT_SIZE,
+        offset + SERIALIZED_UINT_SIZE + len
+    )
     return Pair(payload, SERIALIZED_UINT_SIZE + len)
 }
 
@@ -117,19 +221,31 @@ fun deserializeRecursively(buffer: ByteArray, offset: Int = 0): Array<ByteArray>
         return arrayOf()
     }
     val len = deserializeUInt(buffer, offset).toInt()
-    val payload = buffer.copyOfRange(offset + SERIALIZED_UINT_SIZE,
-        offset + SERIALIZED_UINT_SIZE + len)
-    return arrayOf(payload) + deserializeRecursively(buffer.copyOfRange(offset + SERIALIZED_UINT_SIZE + len,
-        buffer.size), offset)
+    val payload = buffer.copyOfRange(
+        offset + SERIALIZED_UINT_SIZE,
+        offset + SERIALIZED_UINT_SIZE + len
+    )
+    return arrayOf(payload) + deserializeRecursively(
+        buffer.copyOfRange(
+            offset + SERIALIZED_UINT_SIZE + len,
+            buffer.size
+        ), offset
+    )
 }
 
-fun deserializeAmount(buffer: ByteArray, amount: Int, offset: Int = 0): Pair<Array<ByteArray>, ByteArray> {
+fun deserializeAmount(
+    buffer: ByteArray,
+    amount: Int,
+    offset: Int = 0
+): Pair<Array<ByteArray>, ByteArray> {
     val returnValues = arrayListOf<ByteArray>()
     var localOffset = offset
     for (i in 0 until amount) {
         val len = deserializeUInt(buffer, localOffset).toInt()
-        val payload = buffer.copyOfRange(localOffset + SERIALIZED_UINT_SIZE,
-            localOffset + SERIALIZED_UINT_SIZE + len)
+        val payload = buffer.copyOfRange(
+            localOffset + SERIALIZED_UINT_SIZE,
+            localOffset + SERIALIZED_UINT_SIZE + len
+        )
         localOffset += SERIALIZED_UINT_SIZE + len
         returnValues.add(payload)
     }

ipv8/src/main/java/nl/tudelft/ipv8/messaging/payload/TransferRequestPayload.kt

@@ -0,0 +1,38 @@
+package nl.tudelft.ipv8.messaging.payload
+
+import nl.tudelft.ipv8.messaging.AutoSerializable
+import nl.tudelft.ipv8.messaging.Deserializable
+import nl.tudelft.ipv8.messaging.Serializable
+import nl.tudelft.ipv8.messaging.simpleDeserialize
+
+data class TransferRequestPayload(
+    val filename: String,
+    val status: TransferStatus,
+    val type: TransferType,
+    val dataSize: Int = 0
+) : AutoSerializable {
+
+    companion object Deserializer : Deserializable<TransferRequestPayload> {
+        override fun deserialize(
+            buffer: ByteArray,
+            offset: Int
+        ): Pair<TransferRequestPayload, Int> {
+            val (filename, newOffset) = simpleDeserialize<String>(buffer, offset)
+            val (status, newOffset1) = simpleDeserialize<TransferStatus>(buffer, newOffset)
+            val (type, newOffset2) = simpleDeserialize<TransferType>(buffer, newOffset1)
+            val (dataSize, newOffset3) = simpleDeserialize<Int>(buffer, newOffset2)
+            return Pair(TransferRequestPayload(filename, status, type, dataSize), newOffset3)
+        }
+    }
+
+    enum class TransferStatus {
+        REQUEST,
+        ACCEPT,
+        DECLINE
+    }
+
+    enum class TransferType {
+        FILE,
+        RANDOM_DATA
+    }
+}

ipv8/src/main/java/nl/tudelft/ipv8/messaging/payload/UtpHeartbeatPayload.kt

@@ -0,0 +1,18 @@
+package nl.tudelft.ipv8.messaging.payload
+
+import nl.tudelft.ipv8.messaging.Deserializable
+import nl.tudelft.ipv8.messaging.Serializable
+
+class UtpHeartbeatPayload(
+
+) : Serializable {
+    override fun serialize(): ByteArray {
+        return byteArrayOf()
+    }
+
+    companion object Deserializer : Deserializable<UtpHeartbeatPayload> {
+        override fun deserialize(buffer: ByteArray, offset: Int): Pair<UtpHeartbeatPayload, Int> {
+            return Pair(UtpHeartbeatPayload(), 0)
+        }
+    }
+}