Messages part one – constructing the message and headers
Communication in the Bitcoin network is done via messages. A message is no more than just a string of bytes.
This is an example of a simple “ping” message:
f9beb4d970696e670000000000000000080000001b3cb220309941550a2ffd5c # The full message. Header + Payload
The Bitcoin protocol describes how each message should be packet. It is highly important to make sure that you construct the message in the right format. The machine on the other side will expect to receive a very specific format and it won’t be able to process any message that won’t fit this format.
Every message is made of two main components:
- Header (not to be confused with block headers! we’ll talk about them in a later post!)
- Payload
The payload is the body of the message – The message itself. Many types of messages are defined in the Bitcoin protocol, and we’ll talk about each message later on, but for now we should take a look at the header and that’s because the header is the one thing ALL of the messages have in common.
Each header is made of four components:
Size (Bytes) | Name | Data type | Description |
4 | Start string | char[4] | The network identifier |
12 | Command name | char[12] | The name of the command. |
4 | Payload size | uint32 | Len(payload) |
4 | Checksum | char[4] | SHA256(SHA256(payload))[:4] |
Starting-string
The first 4 bytes of the message are the Starting-string (or Magic number).
f9beb4d9 # 4 bytes starting string (Magic number)
This number tells the receiving machine which network I’m using. In this tutorial we’re using the real main-network (Caution! any mistake made in the real main network might cost you real Bitcoins!). The Magic number of the main-network is 0xf9beb4d9. We can look at the “ping” message example at the top and see for yourself that the first four bytes in the message are indeed f9beb4d9. The receiving machine will first check this Magic number to make sure that it is receiving messages from the network it’s currently ruining, and only then it will start processing the rest of the message.
Command name
The next 12 bytes are the Command name.
70696e670000000000000000 # 12 bytes command name
Each header should contain the name of the command, or type of message that is contained in the payload (body) of the message. In our case, the message is a “ping” message. The receiving machine will use this information to know how to parse and treat this message. In our example the receiving machine will answer with a “pong” message (but only after it will complete validating the message). Pay attention that the command field should be exactly 12 bytes long. That means that if the command name is shorter than 12 bytes, it needs to be padded with nulls. We can see that the first 4 bytes makes the word “ping” in hexadecimals (70-P, 69-I, 6e-N, 67-G) and another 8 bytes are nulls (00). In our code implementation (see below) we’re using the function command_padding
to pad our command name to be 12 bytes.
def command_padding(self, command): # The message command should be padded to be 12 bytes long.
command += (12 – len(command)) * “\00” return command
Size
Another 4 bytes will contain the size of the payload.
08000000 # 4 bytes size of the payload
This field is very straightforward. We just need to insert the size of the payload (body) of our message. In our case it’s simply 8 bytes long. (Once again, we need to make sure it will be exactly 4 bytes longs. luckily we’ve predefined this data type in our Bitpy/Utils/dataTypes.py
file under to_uint32(v)
. So we don’t have to manually insert the extra 3 null bytes).
Checksum
The last part of the header is the checksum.
1b3cb220 # 4 bytes checksum
It might seem somewhat strange at the beginning, but all that we need to do is to take the payload of our message, perform the cryptographic function SHA256 twice on that message, and then append the last 4 bytes of the result to our header. SHA256(SHA256(payload))[:4]
def get_checksum(self): check = hashlib.sha256(hashlib.sha256(self.payload).digest()).digest()[:4] return check
Payload
And now all that we left with is the payload – the body of the message . Each message will be parsed differently.
309941550a2ffd5c # The payload, the body of our "ping" message.
The code implementation
In our code we need to deal with both incoming and outgoing messages. So we’ve split our code to two files:
Bitpy/Packets/HeaderParser.py
– to parse the headers of the incoming messages (And after the header was properly parsed, to use the 12 bytes command properly to determine what other steps are required).-
Bitpy/Packets/PacketCreator.py
– to build the header of our outgoing message and to pre-fixed it to the payload of the message.
Bitpy/Packets/HeaderParser.py
for incoming messages
class HeaderParser: def __init__(self, block): # Packets is a stream self.magic = read_hexa(block.read(4)) self.command = block.read(12) self.payload_size = read_uint32(block.read(4)) self.checksum = hash_to_string(block.read(4)) self.header_size = 4 + 12 + 4 + 4 def to_string(self): display = "\n-------------HEADER-------------" display += "\nMagic:\t %s" % self.magic display += "\nCommand name :\t %s" % self.command display += "\nPayload size :\t %s" % self.payload_size display += "\nChecksum :\t\t %s" % self.checksum display += "\nheader Size:\t\t %s" % self.header_size display += "\n" return display
Bitpy/Packets/PacketCreator.py
for outgoing messages
class PacketCreator: def __init__(self, payload): self.payload = payload.forge() # The message payload forged # create the header self.magic = to_hexa( "F9BEB4D9") # The Magic number of the Main network -> This message will be accepted by the main network self.command = self.command_padding(payload.command_name) self.length = to_uint32(len(self.payload)) self.checksum = self.get_checksum() def command_padding(self, command): # The message command should be padded to be 12 bytes long. command += (12 - len(command)) * "\00" return command def get_checksum(self): check = hashlib.sha256(hashlib.sha256(self.payload).digest()).digest()[:4] return check def forge_header(self): return self.magic + self.command + self.length + self.checksum def forge_packet(self): return self.forge_header() + self.payload
Edit (4-Jul-2016): Python 2.5 to 3.5 migration
Please read the general notes about the transition from Python 2.5 to 3.5 over here. And the complete github change log for the migration over here.
The command_padding
function was rewrite in order to take full advantage of Python 3.5 capacity and we’re now using the built in function ljust
.
def command_padding(self, cmd): command = str(cmd) command = command.ljust(12, '\00') return str.encode(command)