Achieve High Throughput Direct File Transfers using NKN client [Decentralized FTP]

NKN client enables free end to end data transmission in a purely decentralized way.

Prior to NKN, if a sender client (a mobile app, for example) wants to send some data to a receiver client, the receiver needs to be publicly accessible (public IP address, open ports), which is not practical for consumer applications, or they have to be both connected to some centralized server or platform, which will introduce additional cost (to build a relay service or pay for the service) and security vulnerability (data exposed to centralized server or 3rd party service).

With NKN client, both sender and receiver and stay private in any network condition, and they don’t need any centralized server or platform.

The data will be routed and delivered in a purely decentralized way, end to end encrypted, and free. This is made possible thanks to the NKN public Blockchain.

NKN client uses a very simple protocol similar to UDP: each packet only contains information about src, dest, payload, and some other necessary information related to signature chain.

A bare minimal NKN client does not create or manage session/stream, and does not guarantee packet ordering, ACK or re-transmission.

The bare minimal protocol works well for some applications, e.g. messaging, real time gaming.

However, if someone wants to transmit large files using NKN client, a few other problems needs to be solved: boosting throughput, packet ordering and re-transmission, etc.

That might sound familiar to you since it’s basically what TCP does.

There are more than one ways to achieve the above goals and the best one depends on the application. As a starting point, we created , a simple application/library based on NKN client that can fast and securely send and receive files between peers without any centralized servers.

It is written in Golang and uses , multiple concurrent clients similar to , and simple TCP-like flow control.

NKN-file-transfer has 4 modes:

• send,
• receive,
• get,
• host.

Send mode should be paired with receive mode to send file to remote peer, while get mode should be paired with host mode to fetch file from remote peer.

Under the hood, both sender and receiver creates multiple clients by adding certain prefix string to the identifier part of NKN address. When a file is being sent, it is first cut into many small chunks (each chunk is 1024 bytes by default) and each chunk is labeled by a sequence number.

Then sender sends chunks concurrently using different path and waits for ACK of each chunk from receiver.

Only a certain buffer size of unacknowledged chunks may be sent at any time (sliding window). If a chunk is not acknowledged after a timeout, it will be resent using a different path.

On the receiver side, when he receives a chunk, he will send an ACK of that chunk back to the sender.

Combining these modes, nkn-file-transfer can be turned into a few quite useful applications:

1. Using send/receive mode, you can send a large or sensitive file to someone on the other side of the planet for free. The speed is quite descent, and content is end to end encrypted, without any other people being able to see the actual content of the file, and everything is decentralized.
2. Using get + send mode on one side (say A) while receive + host mode on the other (say B), A can get file from B or put file to B, just like a standard file server.
3. There is also a HTTP mode that can be enabled together with host and receive mode. It can accept HTTP GET/PUT/HEAD request with route remoteAddr/fileName. It accepts HTTP GET request with Range header so browser can stream video while downloading (the latency is high as you may have expected, but the throughput should be good enough).
4. It can be used as a Golang library to be embedded in other applications.

With nkn-file-transfer, you can often get close to 10Mbps speed to most long range global destinations. We have tested the throughput of nkn-file-transfer by sending a 32MB file from our US office (California) to China office (Beijing) and compared it with WeChat and QQ, two of China’s largest IM that supports file transfer.

The throughput of nkn-file-transfer is 460% of WeChat (which uses a centralized server to store file) and 160% of QQ (which uses a centralized server to relay data but not storing file).

Our nkn-file-transfer is open sourced at . We should address that, it is using a simple flow control algorithm to show how to use NKN client to create high throughput decentralized applications.

We would be happy to see someone achieving better performance using more complicated and throughput-optimized algorithms.

(Featured Image by Yilun Zhang, CTO of NKN.org)