Browsed by
Tag: bitcoin networking

Scripts and stacks

Scripts and stacks

Personal note, Many things happened in past two months the required my full attention. I hope to resume a steady flow of posts in coming days.

Review

In the last post we’ve talked about one the biggest bitcoin misconception – The idea that transaction actually moves coins from one wallet to another. The truth is that transactions are nothing more that statements. These statements always points to a previous statement (that in turn point to an even older statement and so on), and usually these statement also specify an amount of coins that the current owner is wishing to transfer. The statement also contains a riddle, or an equation that needs to be proofed, and mostly, the key to proof this equation will require the use of the private_key that is associated with the recipient bitcoin address.

level3

Pay attention, even though Bob will be required to use his own private_key to proof that he indeed can solve this problem, the private_key still won’t be available to any one.

 

Now let’s look for a second at this transaction message. We’ve already learned how to create a bitcoin message (see this section about Version message and this one about headers). We just need to make sure that all of the fields are filled in accordance the protocol rules. Just like filling a form. You can find a complete list of the fields that needs to be filled in the bitcoin developers documentation.

 

 

Most of the fields are quite straight foreword. I might still create another post in the future with detail instructions on how to fill all the fields, but this isn’t really the topic of this post. This post deals with one of bitcoin more fascinating aspects – The riddle that Alice place in her statement. The riddle that only Bob can solve -The script.

 

(You just can’t wait to create your own transaction? you’re more than welcome to watch my videos on creating bitcoin transaction)

Scripts, what is it?

Scripts is a computer language. In more detail, it’s a set of predefined words that are agreed upon. Every node that follows the rules specified in the bitcoin protocol will know how to read, interpret and implement these words. Because bitcoin messages are basically nothing more then a string on bytes, these words are not written in plain English, rather are translated to OP_CODEs. That way, we can send our message as a string of bytes, and the receiving node will know that these bytes represent some instructions. (Important note, The receiving node will only treat this bytes as instructions only if they appear inside one of the script field.)

Here’re selected few:

Word Opcode Hex Input Output Description
OP_1ADD 139 0x8b in out  1 is added to the input.
OP_1SUB 140 0x8c in out 1 is subtracted from the input.
N/A 1-75 0x01-0x4b (special) data The next opcode bytes is data to be pushed onto the stack
OP_MIN 163 0xa3 a b out  Returns the smaller of a and b.
 OP_SHA256  168  0xa8  in  hash The input is hashed using SHA-256
OP_EQUAL 135 0x87 x1 x2 True / false Returns 1 if the inputs are exactly equal, 0 otherwise.

The original list included around 200 of these words, but currently most nodes will only support few dozes of these words. Using these few words we can create many “riddles” or state many conditions to claim the coins in our transaction message.

For example I can add the following string of bytes as my script.

0x01 0x8b 0x87 0x02 0x87

<1> <OP_1ADD> <2> <OP_EQUAL>
  1. It will take the number 1.
  2. Use the OP_CODE OP_1ADD to add 1 to it -> The output of this OP_CODE will be 2.
  3. Use the OP_CODE OP_EQUAL to make sure if the result is equal to 2. -> The output of this OP_CODE will be True.

A word of caution though, most nodes not only refuse to accept most of these OP_CODEs, they will even refuse to accept most non-standard  scripts, mainly because they want users to use standard transactions. Many nodes will not only refuse to accept a transaction with a non standard script, they’ll also refuse to transmit these transactions to other nodes.

 

Stacks

You might’ve already noticed that this script language can only be written as a list of operations. Unlike other high level languages (such as python for example) Scripts can only be used in a predefined order. This type of structure is called stack, because we’re stacking variables and data on top of each other. But not only we’re stacking them, using the stack structure also means that they’ll be processed in accordance to the order in which they were stacked.

In our previous example, the integer 1 was the first item in our stack. Then came the operation OP_1ADD which took that item as its input, processed this item by adding 1 to it, and than giving the output 2. Now the number 2 is stacked BELLOW the integer 2.

<1> <OP_1ADD> <2> <OP_EQUAL>

<2> <2> <OP_EQUAL>

The node recognize the OP_CODE <0x02> as the integer 2, so it moves on to the next item in our stack – the OP_CODE OP_EQUAL. This operation input is the two items that are directly bellow it and compere the two. If both are equal, it will return True.

<True>

 

This example code can’t be used with a standard bitcoin transaction, it’s only meant to give you a general feel on how scripts works.

You can find an example of a real transaction over here:

 

 

Give it a try with bitpy

One of bitpy newest feature is the ability to create stacks and see them in action in real time. Mind you, only few OP_CODES are currently implemented, but it might still give you a feel on how stacks works.

Example of stack using bitpy
Example of stack using bitpy

 

Simple stack architecture with python

Stack architecture can easily be implemented using arrays. After all, it’s nothing than an array of objects (variables, operations, results etc’).

In our bitpy project, under Utils/OpCodes/Codes.py I’ve created a stack class. In its most basic form, this class will only create and empty array upon initialization, followed by  2 methods only.

class Stack():

    def __init__(self):
        self.items = []

    def push(self, item):
        self.items.append(item)

    def pop(self):
        elm = self.items.pop()
        return elm
  1. push(item) append new item to the array
  2. pop(item) remove the topmost item in my array.

This should be enough to create a very basic stack class. Still, I’ve added few more methods.

class Stack():

    def __init__(self):
        self.items = []

    def isEmpty(self):
        return self.items == []

    def push(self, item):
        self.items.append(item)

    def pop(self):
        elm = self.items.pop()
        return elm

    def size(self):
        return len(self.items)

    def printStack(self):
        display = ""
        for items in self.items:
            items = str(items)
            if len(items) > 5:
                display += " " + "<"+ items[:5] + "..." + ">"
            else:
                display += " " + "<" + items + ">"
        return display

    def clear(self):
        self.items.clear()

The isEmpty method will check if our stack array is empty.

The size method will give us the size of the array.

The printStack will provide us with a visual representation of our array. Pay attention that I’ve limited the size of each item to only 5 characters so that items such as hashed messages, bitcoin addresses, keys etc’ won’t take the all screen.

The clear method will remove all items from our array.

Using this methods we can easily start implementing more advanced OP_CODE to our stack array.

def OP_DUP(self):
    elm = self.pop()
    self.items.append(elm)
    self.items.append(elm)

def OP_HASH160(self): #saved as string!
    self.push(Utils.keyUtils.keys.generate_hashed_public_key_string(self.pop()))

def OP_EQUAL(self):
    elm1 = self.pop()
    elm2 = self.pop()

    if elm1 == elm2:
        self.push(1)
    else:
        self.push(0)

def OP_VERIFY(self):
    top = self.pop()
    if top == 1:
        self.push(1)
    else:
        self.push(0)

def OP_RETURN(self, input):
    self.push(input)

 

Connection part three – Receiving messages

Connection part three – Receiving messages

In the previous posts, all that we’ve done was to construct and send messages to another node on the network. In this post, we’ll see what happens to incoming messages.

First stop – The ReceiverManager:

class ReceiverManager(Thread):
    def __init__(self, sock):
        Thread.__init__(self)
        self.sendingQueue = Utils.globals.sendingQueue
        self.sock = sock
        self.ping = ""

        self.outfile = open("data_received_from_node.txt", 'w')

    def run(self):
        while True:
            try:

                # get only the header's message
                header = self.sock.recv(24)

                if len(header) <= 0:
                    raise Exception("Node disconnected (received 0bit length message)")

                headerStream = BytesIO(header)
                parsedHeader = HeaderParser(headerStream)

                # get the payload
                payload = self.recvall(parsedHeader.payload_size)
                payloadStream = BytesIO(payload)

                self.manager(parsedHeader, payloadStream)

            except Exception as e:
                print(e)
                break

        print("Exit receiver Thread")

The receivermanager always runs in the background, checking our Thread for any incoming packets. Once it receives a packet, it will immediately cut its first 24 bytes.

header = self.sock.recv(24)

The first 24 bytes are the header. If you remember from this post, every Bitcoin message will starts with header, and the header is always exactly 24 bytes long.

 

The first 24 bytes are the header. The rest is the payload
The first 24 bytes are the header. The rest is the payload.

This header is now parsed as a string of bytes and passed to the HeaderParser class in Bitpy/Network/HeaderParser.py

headerStream = BytesIO(header)
parsedHeader = HeaderParser(headerStream)

 

Second stop – The HeaderParser class:

The HeaderParser class takes the first 24 bytes as a long string of bytes, and then it reads them in the same order that we’ve seen before.

Size (Bytes) Name Data type Description
4 Start string char[4]  The network identifier
12 Command name char[12]  The name of the command.
4 Payload size uint32 Len(payload)
4 Checksum char[4]  SHA256(SHA256(payload))[:4]

First 4 bytes for the Start string (or Magic number), another 12 bytes for Command name, the next 4 bytes are the Payload size and the last 4 bytes are the checksum.

4 bytes for starting string. 12 for command name. 4 for payload size and 4 for checksum
4 bytes for starting string. 12 for command name. 4 for payload size and 4 for checksum
class HeaderParser:
    def __init__(self, header):  # Packets is a stream

        self.magic = read_hexa(header.read(4))
        self.command = header.read(12)
        self.payload_size = read_uint32(header.read(4))
        self.checksum = read_hexa(header.read(4))

        self.header_size = 4 + 12 + 4 + 4

    def to_string(self):
        display = "\n-------------HEADER-------------"
        display += "\nMagic:\t %s" % self.magic
        display += "\nCommand name	:\t %s" % self.command
        display += "\nPayload size	:\t %s" % self.payload_size
        display += "\nChecksum	:\t\t %s" % self.checksum
        display += "\nheader Size:\t\t %s" % self.header_size
        display += "\n"
        return display

We’ve also defined the to_string function which basically makes it easier to print a human readable version of the message header.

You might’ve noticed that currently our code just accept the checksum field from the received message without checking it. This is of course a security flaw in our code. The checksum filed is there to help us verify the authenticity of the message. That is one of the ways we can make sure that no one tempered or changed the message on its way from the sender node to our node. But for the time being we’ll assume that the message is indeed authentic and we’ll accept the checksum as is.

 

Third stop – Back to the ReceiverManager:

Now that we have our header, it’s time to get the payload. The size of the payload was defined in the header of the message. We need to cut that amount of bytes from our incoming packets, just as we cut the first 24 bytes of the header. There’s however one extra step in our code. Instead of using the built in sock.recv function (as we did for the header) we’ve decided to implement our own recevall function. The rational was that since we have no way to predetermine the size of the payload, and since the built in sock.recv can’t handle large packets of unknown size, it would be wiser to break the payload into smaller parts and append them together. This has nothing to do with the Bitcoin protocol, it’s only our way to make sure that the code will properly handle large messages.

def recvall(self, length):
    parts = []

    while length > 0:
        part = self.sock.recv(length)
        if not part:
            raise EOFError('socket closed with %d bytes left in this part'.format(length))

        length -= len(part)
        parts.append(part)

    return b''.join(parts)

So now, after we’ve cut the required amount of bytes that represents the payload of our message, and we have both our header (which was already parsed) and our payload (yet to be parsed), we’ll pass them both to the receivermanager manager function.

 

Forth stop – Manager:

    
def manager(self, parsedHeader, payloadStream):

    command = parsedHeader.command.decode("utf-8")
    message = {"timestamp": time.time(), "command": command, "header": parsedHeader.to_string(), "payload": ""}


    if command.startswith('ping'):
        ping = Ping.DecodePing(payloadStream)

        pong = Pong.EncodePong(ping.nonce)
        packet = PacketCreator(pong)
        self.sendingQueue.put(packet.forge_packet())

        message["payload"] = str(ping.nonce)
        self.display(message)

    elif command.startswith('inv'):
        inv = Inv.DecodeInv(payloadStream)
        message["payload"] = inv.get_decoded_info()
        self.display(message)

    elif command.startswith('addr'):
        addr = Addr.DecodeAddr(payloadStream)
        message["payload"] = addr.get_decoded_info()
        self.display(message)

    elif command.startswith('pong'):
        pong = Pong.DecodePong(payloadStream)
        message["payload"] = pong.get_decoded_info()
        self.display(message)

    elif command.startswith('version'):
        version = Version.DecodeVersion(payloadStream)
        message["payload"] = version.get_decoded_info()
        self.display(message)

The manager function does a very simple thing. It checks the command of the message (the command is part of the header) and then it sends the message payload to be parsed by the corresponding functions. For example. If the manager sees that the command is «pong», it will use the decodepong method in Bitpay/Packets/control_messages/pong.py to extract the desire fields out of it. (You can read more about «pong», «ping» and «verack» messages in this post.).

 

Divergence

We have our pared message, both its header and payload. And now we need to decide what to do with them. For some messages this might be the end of the line. There’s nothing more we can do with them. Some might require us to act. «ping» message should be answered by a «pong» message, transactions should be checked and relayed (We’ll talk about transactions in later posts), «version» messages should be acknowledged by sending back a «verack» message.

A major part of learning the Bitcoin protocol is learning how each and every message should be dealt with. Which fields of information it contains and what is the meaning of this information. We’ve already talked about some of the messages in previous posts  (see here for «ping», «pong» and «verack» messages, and here for «version» message.) and as our project will have more features implemented, so we’ll discuss other type of messages and how to deal with them.

Connection part one – Finding a node and packets routing

Connection part one – Finding a node and packets routing

The first thing we need our code to do is to connect to the Bitcoin network. this is relatively straightforward process, we just need to find one node in the network and establish connection  with that node. A list of few of the active nodes can be easily found online. We’ve randomly picked one node from this list on blockchain.info .

We’re using the socket module to establish our connection using this simple code:

import socket
import sys

HOST = "66.90.137.89"
PORT = 8333

"""
    We will use this file to connect to one node
    But in the future we will connect to more than one
"""

def connect():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    try:
        sock.connect((HOST, PORT ))
    except Exception as e:
        print e
        sys.exit(0)

    return sock

 

we’ve also created main.py at the root of our folder structure (right under Bitpy/). This will initialize the connection code upon startup and will route our incoming and outgoing packets to ReceiverManager  and SenderManager respectively.  The queue module helps us to make sure that the packets are being processed in the right order. We’ll later  see what each file does, but for now, what is important to understand is:

  1. We’re connecting to another node on the network.
  2. We’ve found the address of this node on a public list at blockchain.info.
  3. The connection code is stored at Network/connection.py
  4. We’ve created a Main file under our root directory (Bitpy/Main.py) that will initialize the connection to the node, and will route our incoming and outgoing packets to one of the two queues  files SenderManager.py (for outgoing packets) and ReceiverManager.py (for incoming packets). Both files can be found under Manager/.
  5. The user manually specify which packet (message) he wants to send using the core_manager. We’ll talk about it later on when we’ll be dealing with the user interface.

 

So now we should have a look at our Receiver/Sender Managers, but our ReceiverManager is a bit too complex for this stage, so we’ll talk about it later, once we’re ready to talk about parsing incoming messages. For now, we’ll only have a look at our SenderManager.

The first thing we did was to use the threading module. This module allows us to keep our connection asynchronous, that means that we can receive and send messages at the same time. Apart from this threading module this file contains only one more class – SendingManager. Once this class is defined, it will have access to our thread, it will be able to use or sock object (declared in connection.py) to connect to the remote node and it will also receive the packets queue from the main.py file.

 

from threading import Thread


class SenderManager(Thread):

    def __init__(self,sock, queue):
        Thread.__init__(self)
        self.sock = sock
        self.queue = queue

    def run(self):
        while True:
            if not self.queue.empty():
                order = self.queue.get()
                self.sock.sendall(order)

        print "Exit sender Thread"

 

So the main.py file gets a list of packets (messages) from the user which he wishes to send. (The user creates the packets in the core_manager.py file). The packets are stored in a queue, and a SenderManager object is then created. It gets access to the sock object, the thread, and the queue , then it will simply send the packets in their order, as specified in the queue, one by one, to the ip address and port of the sock, while making sure that the connection remains asynchronous.

 

Before we can start sending and receiving messages, we first need to learn about messages.