Browsed by
Tag: Bitcoin

Get your Bitcoin address using Ethereum smart contract

Get your Bitcoin address using Ethereum smart contract

Ethereum and Bitcoin are both using the same type of encryption, the ECC (Elliptic Curve Cryptography) over the same graph (256k1). While it’s not really recommended, the same key pair can be used both for Bitcoin and Ethereum.

A simple Solidity code can be used to get the Bitcoin address of a public key. Such a code can run locally (as a constant function) on the Ethereum Virtual Machine to save gas, or as a regular Ethereum transaction.

The code in this example requires the user to insert their public key in its uncompressed format as an input; then it produces the binary address that matches that uncompressed public key for the main Bitcoin network. The code can be easily modified to work with compressed public keys as well (just remove the yPoint and add the side of the ECC graph). The code can also be amended to give the binary address of other testnet/namecoin.

 

How to create a Bitcoin address

The most basic process of deriving Bitcoin address from a public key is set in the following technical documentation.

 As you can clearly see, there’s very little to it than just hashing and appending.

Recreating the process in Solidity

First, let’s generate a random keypair using bitaddress.org. Under the tab “wallet details” we can see the uncompressed public key.

The public key
xPoint = C4BB8E42F7DA5504A456C16BE533549DA4FE580279382478F3365FF7CCBF032D
yPoint = 68A73547E809F1ABFAA51D10019E8AC682D1205448042326E9E3B91841CB9FA7

Now let’s create our smart contract in Solidity:

pragma solidity ^0.4.11;

contract BitValid{
	
	bytes32 constant mask4 = 0xffffffff00000000000000000000000000000000000000000000000000000000;
	bytes1 constant network = 0x00;


	function getBitcoinAddress(
			bytes32 _xPoint,
			bytes32 _yPoint)
			constant
			returns(
				bytes20 hashedPubKey,
				bytes4 checkSum,
				bytes1 network)
	{
		hashedPubKey 	= getHashedPublicKey(_xPoint, _yPoint);
 		checkSum 	= getCheckSum(hashedPubKey);
 		network 	= network;
	}

	function getHashedPublicKey(
			bytes32 _xPoint,
			bytes32 _yPoint)
			constant
			returns(
				bytes20 hashedPubKey)
	{
		var startingByte = 0x04;
 		return ripemd160(sha256(startingByte, _xPoint, _yPoint));
	}

	function getCheckSum(
			bytes20 _hashedPubKey)
			constant
			returns(
				bytes4 checkSum)
	{
		var full = sha256((sha256(network, _hashedPubKey)));
		return bytes4(full&mask4);
	}
}

The function getBitocinAddress() takes the x and y coordinate of the public key from the user, both are 32 bytes long (the uncompressed public key) and will return 3 variables, the hashed public key (bytes20), the checksum (bytes4) and the network starting byte (bytes1).

The network starting byte is currently hard codded to 0x00 (the main starting code). You can change this code to work with any other test network.

The hashed public key is obtained by hashing the public key (both x and y coordinates) with the starting byte 0x04 twice (as described in the technical documentation). Once with sha256 and then again with ripemd160. The finale result is 20 bytes long.
function getHashedPublicKey(
		bytes32 _xPoint,
		bytes32 _yPoint)
		constant
		returns(
			bytes20 hashedPubKey)
{
	var startingByte = 0x04;
	return ripemd160(sha256(startingByte, _xPoint, _yPoint));
}
After we got the hashed public key, we’ll prepend the network byte to it and hash it again twice using the sha256 function. The result of 32 bytes long is used to construct the checksum, a special 4 bytes that are used to allow another user to verify that the Bitcoin address they’re sending to is indeed a valid address.
bytes32 constant mask4 = 0xffffffff00000000000000000000000000000000000000000000000000000000;

function getCheckSum(
		bytes20 _hashedPubKey)
		constant
		returns(
			bytes4 checkSum)
{
	var full = sha256((sha256(network, _hashedPubKey)));
	return bytes4(full&mask4);
}
We don’t need all of the 32 bytes, only the first 4 bytes, but slicing variables is a hard thing to do in Solidity. Luckily, Solidity does allow for easy bit manipulation and masking. You’ll need to create a mask of 32 bytes to match the 32 bytes of the sha256 output. This mask should take only the first 4 bytes, as they’re the real checksum.
The full result (32 bytes) = 0x4c30ed507a508af52063560ff8f1c09e66be0587868a0b8ca21ab337440e4e8e
Mask for the first 4 bytes = 0xffffffff00000000000000000000000000000000000000000000000000000000
checksum = 0x4c30ed50

The results

At the end of the day, we have the following three components to return to the user, the network byte (currently hard coded), the hashed public key and the checksum. These are the three components that make up a Bitcoin address.

However, this isn’t the last step. In Bitcoin, a special type of encoding is used called base58. The current code doesn’t convert the result into base58 (I’ll leave it for another day), so we’ll be forced to do this step manually.

The following website provides some tools to convert our bytecode into base58. This is basically the final Bitcoin address.

At the end of the day

Using Solidity to retrieve the Bitcoin address that matches a specific public key (and therefore, a private key as well) might be useful when you’re trying to create a smart contract that maps some events between entities on both blockchains and I suspect might have some value when dealing with identities. The procedure isn’t cheap on gas but can be done locally using the EVM. It’s a shame that there’s no access to the bytecode of the transactions in Solidity since it could have made the process of finding the Bitcoin address of the message sender automated.

The blockchain developer path – Princeton University – Bitcoin and Cryptocurrency Technologies

The blockchain developer path – Princeton University – Bitcoin and Cryptocurrency Technologies

Mapping the blockchain education ecosystem

When I created my first tutorials almost two years ago, there were very few educational resources about blockchains. The ecosystem was sorely lacking in good courses, tutorials and guides to ease the learning curve for newcomers. I remember the many days I’ve spent reading raw codes and technical documentation until I was finally able to manually connect to the Bitcoin network, create keys, sign transactions, etc.

But two years is forever in the world of blockchain, now many companies, universities, and individuals are flooding the market, offering their services as educators and validators.

The thing is that this ecosystem is still kind of a wild west. There are very little standardization and collaboration between different players in this ecosystem. And the final result is sub-optimal, both for those who wish to educate themselves and for those who want to work with them (future employers/potential business partners).

But fear not, I’ve taken upon myself to try most of the major courses available and to receive as many certifications as possible. And I’ve got some interesting insights on the right route an aspiring blockchain developer should take.

I the following posts I’ll review the following courses and certification process:

  • Princeton University – Bitcoin and Cryptocurrency Technologies. Available as a full course at Coursera
  • Berkely University – BLOCKCHAIN at Berkely.
  • Stanford University – Bitcoin and Cryptocurrencies.
  • Multiple Youtube channels and videos (including my own)
  • Book – Mastering Bitcoin: Unlocking Digital Cryptocurrencies by Andreas Antonopoulos (1st Edition)
  • Book – Bitcoin and Cryptocurrency Technologies by Princeton University (Draft version)
  • Udemy Ethereum: Decentralized Application Design & Development
  • Diginomics – Ethereum developers course (Created by myself more than a year ago)
  • IBM – Blockchain Essentials for Developers (Includes certification)
  • C4 – Certified Bitcoin Professional.

Princeton University – Bitcoin and Cryptocurrency Technologies

The oldest one still open to the public

Princeton University created one of the first blockchain courses out there, and because of that, they are the first one to be covered in this series. The course is offered since early 2015 (Some of the videos were created in mid-2014). The course is instructed mostly by Arvind Narayanan, Assistant Professor at Princeton computer science department, with few guest appearance by Joseph Bonneau, PostDoc at Princeton, Prof Edward W. Felten, Professor of CS and public affairs at Princeton University.

The course is publicly available on the Coursera platform, and while it might seem as if there is a time limitations, a new class starts every week. Princeton also offers to the public its book “Bitcoin and Cryptocurrency Technologies” (the draft version is free) that follow up the course nicely.

The course is divided into 11 weeks. All weeks are available since day one and there’s nothing that stops you from going through all of the course in one go. Each unit contains few videos, with a simple quiz at the end of each video. You don’t have to solve the quiz to complete that lecture. You’re free to watch the lectures in any order you might want. You don’t have to watch all the videos in order to complete the course.

In order to complete the course, you’ll need to complete three programming assignments.

Course structure

The first three weeks are dealing with the basic aspects of Bitcoin and the blockchain.

It starts by exploring the hash function, and continue to data structure where the hashes are keys. The concept of DAG is hinted in the videos but isn’t entirely covered although Merkle trees are mentioned. A simple Introduction to Asymmetric (Key) cryptography is given but in a very shallow way. ECDSA is also mentioned – ECC is almost completely ignored. RSA isn’t even mentioned. The first week ends with a demonstration of mock cryptocoin (Scroogecoin, we’ll get to that soon).

In the second week, we’re giving a simple introduction to some concepts of decentralization and return to hashes in the context of POW.

The third-week deals with transactions in Bitcoin. The first couple of videos deals with P2PKH and P2SH transactions. The first video also introduce many concepts that are relevant to Bitcoin transactions (UTXO, scripts, inputs, etc.) Scripts are then explored in greater detail. I find the scripts part to be quite well explained. This week ends with an introduction to the concept of blocks and the basics of networking architecture in bitcoin.

All in all, I found the first week to be the weakest in this batch. While the jargon used is aimed at people with background in computer science, many terms were merely glossed over. It made it hard to determine who the course is aimed at – beginners? Individuals with background in computer science? Mathematics? Also, the first assignment (Scroogecoin) requires the students to continue watching all the lectures in week 2 & 3 before being able to solve it. I believe that students taking the course for the first time might try to complete the assignment before moving on, which is another source of frustration.

Week number 4 is kind of stand by itself as it deals mostly with how to store and use bitcoin by looking at wallet types, exchanges and so on. The jargon in this part is more mathematical/technical then week 2-3. Some of the material feels somewhat outdated, but other parts like the way to split and share keys and fees calculation were quite simple and satisfying. However, seeing as most topics during this week might appeal to the more general user, one with little to no background in CS or another related field, I find the return to the more technical jargon being quite counterproductive.

Week four also ends with another assignment. As before, this assignment can’t be solved just by watching the lectures, but I guess that by now most students have already realized it, and won’t postponed watching the rest of the lectures for a time after they’ve managed to solve the assignment.

Week 5 – 11 deals with many different aspects of Bitcoin. Mining, anonymity, legal, and bitcoin as a platform. Each lecture is good by itself as a mean of introducing the students to the basic concepts, but it never goes further than that. In that regard, I see little value in following the structure of the course. It would have been better to just provide the students with a simple glossary/list of topics to cover with links to external resources. Still, some of the lectures were highly enjoyable. In this units I’ve enjoyed the lectures on mining algorithm, which starts to show a simple introduction to the way the hashing functions work, the zerocoin introduction was also useful in providing the foundations for newcomers. Bitcoin as an append-only log especially resonance with my ideas on the blockchain and its use case.

Week 7 (deals mostly with the inner politics of Bitcoin) ends with the last assignment that the students are required to submit. Once again, the assignment has little to do with the lectures themselves.

The assignments:

As mentioned before, the course contains three assignments. All three are in Java and deals with validating a transaction, validating a block, and reaching consensus.

This three task represents a substantial part what is required from a bitcoin node. All three are graded by the use of very impressive test engine that runs multiple test cases on your code and validates the results. The amount of effort placed in creating these assignments is visible and because the course is giving for free, online, and bears the Princeton University name with it, makes these assignments even more valuable.

Which makes it even harder for me to say the following. While I can appreciate the effort putting into these assignments, they feel much more like homework on Java and OOP with a Bitcoin “theme” than a real Bitcoin coding exercises.

The first assignment requires the students to prove some understanding to the way Bitcoin transactions are validated and chained to each other. This is the most Bitcoin-related assignment of them all as it requires some understanding of how bitcoin transactions work. Still, at the end of the day, the real task here is to create an algorithm that will return a specific set of valid transaction and it feels as if a substantial part of the time placed into solving this assignment is spent on recursive loops and working with sets and hash tables rather than working with more aspects that are unique to bitcoin.

Assignment 2-3 were completely off. The second assignment required the students to reach a consensus between nodes in a way that has nothing to do to with how blockchain works. This assignment feels like a pure algorithm task. It’s clear that the creators of the assignment thought carefully about it and worked hard to create fair and interesting tests. After all, more than one algorithm can be implemented to reach a consensus. But at the end, it turns out that the task can be solved by using a ridiculously simple algorithm. I could almost imagine the creators of this assignment slap themselves on the back for managing to create assignment so well though that it can be tested and passed by multiple algorithms with a variety of complexity levels. It is impressive, but have very little to do with the way bitcoin works.

Assignment three starts to focus again on Bitcoin. It requires the students to validate blocks and constructing the blockchain. This assignment is somewhat more realistic and more connected to the way Bitcoin works. But it still requires the students to spend to much time constructing classes, methods, loops and sets – and too little time to really understand how the blockchain is constructed.

The assignments can be submitted as many times as you like, and I highly suggest to try and submit even half-baked codes. The automated grader will run the test engine on your code and might provide you with some interesting insights regarding your codes. For the “less honest” students out there, the code for the test cases (and more) is just a short google away.

The grader engine provides useful information. You can resubmit your files as many time as you like

The course forum

The course has a forum which goes all the way back to 2015. It’s not limited just to the current course timeframe. In this forum, some interesting discussions are taking place, and I highly suggest to join the course if only to read some of the conversations taking place there. These forums also contain a lot of information regarding the assignments which I also suggest looking into.

Graduating and certifications

The course can be completed by submitting the three assignments and receiving a grade of 60 and above. There’s no need to watch all the videos or taking the quiz at the end of each video to complete the course. The assignments can be submitted as many time as you like so that you’re giving the chance to improve your grades.

Upon completion, you’ll receive an email congratulating you for completing the course. You can also verify your identity at Coursera by uploading your current photo plus an office identification (passport or driver license). Once Coursera verifies your identity, you can share your achievement online on Facebook, Linkedin, etc.

The certification issued by Courera and doesn’t bear the Princeton Univesity name. Knowing that the course is free, the assignment has little to do with Bitcoin (and can be easily cheated), this certification carries little to no weight.

You can share your achievement online. No other recognition is provided

Final thoughts

The beginning of the course (weeks 1-3/4) is its the weakest part. Students who want to understand how transactions, keys, hashing, POW, etc. works, will be better to look for it in other places. After that, the course becomes somewhat less technical. This is a welcome change as it allows it to provide a basic introduction to many different aspects of the blockchain.

In this regards, when ignoring the first few weeks, this course might appeal to students who already understand some of the basic concepts of the blockchain and Bitcoin and are just looking for a good source to review as many aspects of it. Each video is no more than 20 minutes. And the list of topics that are covered is quite impressive.

The assignments are wonderful as an exercise in Java and OOP but provide very little extra value for those who are interested in the blockchain. It’s obvious that one can pass the course just by utilizing his/hers experience in OOP while having little understanding of the Bitcoin works.

Completing the course can be achieved in a single day if one chooses, and being able to share your achievement online is a nice bonus, especially when Princeton University is mentioned.

Take this course if:

·       You understand Java and OOP

·       Have some backgrounds in how bitcoin works regarding transactions, keys, blocks, etc. and want to enrich your knowledge of other aspects of it

Time to complete the courseYou can complete the course in one day or eleven weeks. It’s up to you
InteractionThe course forum contains some interesting conversations
Materials

·       Videos mostly

·       The draft of the book Bitcoin and Cryptocurrency Technologies si also available

·       Three programming assignments

CertificationYes, limited. Provided by Coursera for those who complete the three assignments
What to teach when teaching blockchain

What to teach when teaching blockchain

This article is part of a series of articles depicting my experience with creating and conducting an 8 week long blockchain app development course in Brazil.

 

The course as a mega structure

After I accepted to take on this challenge, the next logical course of action was to create the listof material I intended to teach the students. Originally, this list contained almost everything blockchain related – from bits, bytes and creating protocol messages all the way up to an overview of the history and politics of Bitcoin. The list was long, perhaps too long. I must admit my own limitations- I couldn’t see any realistic way to create such an extensive course by myself.

In addition, I had some doubts as to how many students can actually properly digest so much material, even when given 8 full weeks. (6 actually, if you count the time given to personal projects). I realized the list needed to be more focused – and that’s where my first major decisions were made.

 

Decision number 1: Focusing on the protocol itself. By deeply understanding the Bitcoin (and later the Ethereum) protocol, I believed the students will gain some clarity and very strong foundations upon which they can later build on and learn more extensively by themselves.

It was highly important to me to ensure they actually understand the protocol itself – how it developed, the tools it utilizes, and the logic behind choosing these specific tools and architectures and so on.

Once the students understood the intricacies of dealing with it, they will be better equipped to learn by themselves other things like how to use JS libraries in web applications; read and understand whitepapers and BIPs/EIPs, and create their own private chains and more. But how do I teach the protocol? That’s where my second decision was made.

 

Decision number 2: Bitcoin is the gold standard of blockchains and therefore, the go to chain when learning the basics of blockchains. This decision was a no brainer for me and represents a substantial chunk of my personal view on the blockchain ecosystem. The Bitcoin blockchain is the most widely used, heavily researched and extensively documented (albeit the documentation is somewhat lacking in my opinion). Every blockchain in existence this day stemmed from it and relies to a large extent on many of the concepts introduced by it.

It was clear to me that creating a blockchain course that isn’t based on the Bitcoin protocol will be a major breach of the faith the students reposed in me.

These two decisions brought with them peace of mind. Whereas before I was drowning in a pool of topics to cover – a list so long and chaotic, it was completely unmanageable, I now had a solid anchor to work with.

 

 

The four stages of mastering the blockchain

Once the list was completed, it became clear it could be divided into 4 main stages or phases:

The basics

From bit and bytes to protocol messages. The idea is that once completed, the student is able to send, receive and (maybe most importantly) parse a Bitcoin protocol message. In order to achieve this, the students will need to learn the basic mechanism of networking, create the byte code for a Bitcoin message – both payload and header, learn how to use one-way functions, hashing and how to read and understand the Bitcoin developers guide and protocol documentation.

Here, I held enlightening conversations with the students that revolved around keys. We went through the documentation together in a bid to learn about how to move seamlessly from private key to the public one and onto Bitcoin address.

Transactions

The bread and butter of the Bitcoin protocol. Now that the students have the understanding of how to work with the Bitcoin documentation, the next logical step was channeling that documentation and using it in creating transactions.

Here, the students will garner all their knowledge about bits and bytes, hashing and keys (from the previous stage) and channel it into creating the raw Bitcoin transaction. After which, they will learn about the pkScript; scripting in Bitcoin (and scripting in general) plus stack architecture and how the pkScript is executed, which by the way is also a great opportunity to start the conversation with the students about Ethereum and its virtual machine.

 

Finally, the students will go on to elucidate on what type of transactions are considered standard.

 

Blockchain architecture

After speaking with the students about all the basic components required in creating and transmitting a Bitcoin transaction, it was time to move on to talks about blocks and their architecture. I wanted them to have an understanding of what blocks really are. We were to create merkle trees (and merkle roots), blockheaders, calculate (mock) difficulty and target, and find the proper nonce. Along with that, it was also a great chance to have a chat with the students about DAG and consensus, mining algorithm, forks and altcoins.

In this phase, I was also afforded an invaluable chance to have a discussion with the students about contracts and secondary layers such as lightning network and BIPs (SegWit, addresses and other bips).

Ethereum

This is the second most predominant chain out there. It includes the wild west of consensus rules, mining algorithms and script (VM) playground. Many students want to understand how it works, and rightfully so. However, what most students want in learning Ethereum is how to actually use it – how to write smart contracts, and hence, how to use Solidity. This is where my previous decision to focus on Bitcoin as the gold standard for blockchain architecture became two opposite things; a blessing and a curse.

I wanted to spend as much time as I possibly could on Solidity, Web3 and Ethereum specific tools, and since I already spent more than half of the course talking about blockchain and Bitcoin, I was freer to talk about Solidity and web3 in this part of the course. However, a compromise was made and several unique architectural aspects of Ethereum were left out. I briefly talked about gas and the EVM by tying it to Bitcoin scripts and fees, and I glossed over block structure by simply stating that “merkle roots are also involved”. Mining and Ethereum attempts to move to POS were completely ignored – I think you get the picture.

The only architectural aspects of Ethereum I discussed in class were the ones that can be easily seen when playing with Solidity and smart contracts (Libraries and calls, Inline assembly, variables gas price, constant functions, variables scopes etc.)

 

I used this general scheme to select the topics to be covered in the Bitcoin part of the course

 

A sense of achievement

All in all, I’m quite proud of this structure and the topics it included. I found myself with a solid and detailed road map of the course. It was clear to me and to a lesser extent my students where we are now, and what is yet to come. I also hoped to use these 4 stages as the basis on which I would create semi-official certificates for my students. The idea was that every 1.5-2 weeks or so, the students did manage to achieve something very meaningful.

On completion of the first stage, the student managed to establish connection to a remote node on the Bitcoin network, and created key pairs and Bitcoin address.

On completion of the second stage, the student proved that he/she possesses a deep understanding of how Bitcoin transactions work. Things like how to hash a transaction, how to deal with TXID, how fees are determined, how (and when and where) signatures are used in the protocol and how the Bitcoin scripting language works. All of these gave the students a platform to create the bytecode for their own transactions (P2PKH and OP_RETURN).

At the time the third stage was completed, the students had managed to create their own block by creating the merkle tree and finding its root, calculating (mock) target, creating the blockheader and eventually finding the right nonce (Important note: I didn’t ask the students to create a coinbase transaction, although we did talk about it and examine its structure, but I believed adding another type of transaction will only serve to confuse them).

The final stage, the Ethereum stage, was in fact more of free working sessions. As I mentioned earlier, one of the requirements was that the students will have time to create their own dApps. Each student was working on his/her individual projects and as long as they managed to get their smart contracts to work, they managed to complete this stage.

(More details on my experience with the certification system in the coming articles).

 

 

So how did it go?

The first part of the course went rather smoothly, either because the students were still very eager to learn or because it was truly well constructed.

However, owing to some technical problems, I did not receive the certificates on time which placed me at a somewhat awkward position. Not only did I make a promise to my students that I’d give it to them, I was also quite worried that in the absence of something more tangible, they might not properly appreciate their own progress during the course.

This fear continued to be my shadow through the duration of the course as I tried to motivate my students to continue their hard and not so rewarding work. Right at the end of the course, I did manage to receive some documentation I could give my students and the response was very positive indeed. I’m now more convinced than ever that providing a proper certification at the end of each step can reduce much of the stress and hardship associated with such a demanding course. As a matter of fact, one of the major decisions I made at the end of this course was regarding certifications.

There was a major criticism I received during the course in relation to the course structure. A good number of the students, while not showing any major problem understanding the granular topic of the day and implementing the codes did have a hard time seeing the bigger picture. The structure I created aimed to look at the blockchain from a very close distance while working with some of its finest components.

I did try to encourage additional “free conversations” in class, hoping that during these less formal chats the students will utilize what we learned But soon it became clear to me that the timing of these free conversations, their length and their focus (or lack of it), was quite counterproductive. Therefore, I needed to come up with more appropriate time slots for such informal class sessions, while making sure to be well prepared with case study in case the students will not be focused. . For example, it might have been productive to try and talk about SigWit during the transactions class instead of only at the end of the Bitcoin portion of the course, even though the students didn’t yet learn about merkle trees, blocks architecture and BIPs.

The students were vocal about their desire for me to focus more on the big picture. This was in addition to some external pressure placed on me to start working with the students on their own personal projects. All of these led me to the production of some minor (yet quite meaningful) changes to the structure, mid – course. Basically, the border between the third stage (Blockchain architecture) and the fourth (Ethereum) was less strict than I wanted it to be.

It is true I already planned to use the third stage as a transition stage (as mentioned above, a substantial part of the Ethereum architecture was glossed over by comparing it to the Bitcoin blockchain). I did plan to talk about Ethereum quite extensively at this point (Ethereum was supposed to be mentioned first at the end of the second stage- transactions stage- right when the students learned some standard pkScripts). I still wasn’t well prepared with a suitable road map to make this diversion from the predefined curriculum.

The final result was that the transition from Bitcoin to Ethereum was lacking at best, if not somewhat confusing. This experience, while being somewhat hard in real time, did help to solidify my own confidence in the structure of the course – at least in its higher level and I’m much less inclined to perform similar deviations in future courses.

Another veritable source of confidence is the progress many students showed. A small, yet substantial, part of the class came on board with little to literally zero experience in programming and blockchains. Yet, by the end of it, I did witness some highly impressive progress in their abilities and understanding while the more advanced students also managed to create some impressive codes, smart contracts and even to construct a few apps. Ultimately at the end of the day, that’s what matters the most.

Teaching blockchain in Brazil

Teaching blockchain in Brazil

What is it all about

A few months ago, I received an interesting email from a company I didn’t know at the time. What was the content? I was asked to create and conduct a full fledge, 8 weeks long, blockchain development course in Brazil that would involve students from all walks of life.

Blockchain education is something I am passionate about, so this offer immediately struck a chord in me. In the past, I have talked a little on how I got into this field and given some insights on my views about the current ecosystem in terms of education. To sum it up; there’s an existing huge knowledge gap that serves to keep many talented people from properly contributing and utilizing numerous blockchain based solutions.

The deep-seated desire to bridge this gap is a major part of my personal agenda. For that reason, in the past, I created a few videos and tutorials (which owing to the fast paced nature of technology are in sore need of an update, I know) that were aimed at mitigating this gap in knowledge and making blockchain codes somewhat more reachable for developers and tech people who are new to this ecosystem. In addition, I was looking for other ways to bring Bitcoin and Ethereum to the people. In line with this, a creating/conducting course seemed like the logical next step.

In the early chats that transpired between me and the Brazilian company, I was offered the freedom to construct the course as I see fit, without any restrictions.  There was one specific request – that when the duration of the course comes to an end, the student will create their own apps. Considering the proposed length of the course (8 weeks), I believed  it would be quite easily achievable simply by allocating the last 2 weeks of the course to exclusively working with the students on their personal projects.

As we talked about the projects the students might create during the course, we also discussed the lack of needed quality in many of the current blockchain related projects out there.  I gave my impression that it was due to a dearth of understanding of blockchains. Most projects are nothing more than basic apps that have very little to gain from using the blockchain and are either utilizing it for marketing reasons – basically to draw more investors’ money owing largely to a lack of understanding on the investors’ side.

But this is not all; it could also be as a result of a serious flaw in their understanding of how to use the blockchain from the creators’ side. I wanted to ensure the students really have an understanding of what the blockchain is, and equally important, what it isn’t.

My counterpart was impressed by this approach and it became clear to the both of us we were seeing eye-to-eye on what our vision for the course was.

 

Getting ready

Working on the course took a substantial part of my time for the better part of 2-3 months. I basically started creating a course from scratch as there were very limited resources and existing courses to draw from. I created a course structure, list of topics, codes and other teaching aids. The task was a hard one, harder than I initially anticipated. I basically, single handedly tried to create one of the most ambitious courses using nothing but my own means and with literally no support of any kind. But I made it!

Slowly but surely, I managed to create a list of working codes to teach my students. I had a solid course structure, the basic infrastructure for a certification system, few assignments, list of resources, presentations and a lot of notes – all corresponding and completing each other, tested and organized for maximum effect.

 

In Brazil

It was finally the moment of truth. I arrived in Brazil with my course (mostly) neatly organized and prepared. The time for theory was past and it was  time to see it in real action and put my work to the ultimate test.

 

Some curious Brazilian horses that participated my class.

 

The final experience in Brazil was much more challenging than I thought it would be. Some of the difficulties I encountered were related to the technical background and level of the students (the class was very heterogeneous). While some had extensive experience, others had absolutely zero previous experience.  A number of the challenges were related to my own preparation for the said course.

Still, when the course winded to an end, I was highly pleased and proud to see how ALL of the students managed to show great advancement.

I’m very grateful for this experience. I had the opportunity to create a full fledge course, a unique and solid one that provides substantial blockchain education, an element that is sorely lacking outside of very specific programs in selected universities. The challenges I encountered during the course were also a great blessing; my course (and by extension, my whole approach to blockchain education) was tested rapidly and almost every aspect of it was subjected to stress test, and it made it through!

Additionally, this course was a great learning experience for me. As a matter of fact, I’m already working on implementing some of the things I’ve learned in order to provide even better learning resources and courses in the near future.

Now, almost a month after the course, I decided to sit down and articulate my thoughts and experiences. It is my wish to share what I did and what I learned. It is my hope that by doing so, I might be opportune to effect some positive impact on the blockchain education ecosystem and that these articles will be helpful to many.

The articles are currently divided into 7 major categories:

  1. Course structure.
  2. Choosing the working environment.
  3. The codes that were presented and exercised.
  4. Certifications and accreditation.
  5. Teaching aids.
  6. Assignments, homework and evaluation.
  7. Others (students’ background, learning environment, marketing and setting expectations, class size, after class meetings etc.).

These articles will be published in parts in weeks to come.

 

 

blockchain related career

blockchain related career

A pleasant call

A few months ago I was approached by the wonderful guys and girls from the Exosphere Academy with the offer to help them build a new blockahin course. They wanted to create a boot camp for people who are interested in blockchain, its implications and possible implementations. I liked the idea. For some time now I believed that one of the major obstacles for bitcoin, ethereum and other blockchain utilities to achieve their potential lies at the high learning curve new users and developers needs to climb. In this blog, and in other forums, I did my best to try and mitigate this learning curve and I was honored by the opportunity to construct a complete and cohesive course for other developers and enthusiasts.

 

My way.

After I took upon myself the task of creating this course, I was approached by a Brazilian magazine who asked me to describe how I became a blockchain developer. The result was the following article at carreirasolo.org. (The original article is in Portuguese). And right here is an English translation for all of those who’re interested in blockchain related carer.


 

The first time I heard about bitcoin.

Six years ago, I read a short article in a financial magazine about a new type of digital coin called bitcoin. Naturally, the article was completely misleading, which isn’t that surprising. At the time, nobody really understood bitcoin. I’m not even sure the article mentioned the word “blockchain”. But I will always remember this article and how I just skimmed it and completely disregarded the idea as just another scam – thus forfeiting my chance to become a millionaire by downloading a piece of software to my computer. A few years later, in 2012, when I was (slightly) older and wiser, I read another article about bitcoin, this time in a tech magazine. Although it wasn’t completely accurate, the article provided a great service to its audience. The reporter was not only informed about bitcoin, but he also put a lot of effort work into understanding his readers – their technical background and how well informed they were about economics, web architecture, computer science, etc. The result was the first ever bitcoin article I read that dealt with it in terms, analogies and examples that conveyed the actual ideas behind the coin.

After that, I was hooked. The protocol’s promise is one of the most ambitious projects I’ve ever encountered in the digital world. It’s not just another cool app, video game or social network where we can show off pictures to our friends and family. Bitcoin challenges us to literally rethink, remodel and rebuild some of the most central elements of human society – from financial systems, government bodies, bureaucracies, health care, sources of information and much more. All of these can benefit from implementing at least some of the tools presented in the bitcoin protocol. I knew that I wanted to be a part of it.

Baby steps

Initially, I didn’t have much coding experience. At the time I was finalizing my BSC in chemistry and environmental studies. I wrote some very basic Java code to help me in my work, but nothing more than that, so I knew I had to work on my coding skills – sooner rather than later. I decided to study Python for many reasons: it was easy to use, well documented, and already had a (relatively) large selection of bitcoin-related libraries and implementations. I was actually amazed at how well it went. Python turned out to be an extremely easy language to learn, but what was even more encouraging was that while learning the basic concepts of the programing language, I also gained many valuable insights into computer architecture and protocols. All this was very useful when I tried to figure out how Bitcoin worked.

After I felt reasonably competent with Python, I decided to look more into app development. Luckily, at the time, many JavaScript bitcoin-related libraries started to appear. JavaScript was easy enough to learn (especially when all you care about is the bitcoin part, and not the user experience). HTML and CSS were somewhat more confusing, but I soon managed to create some decent-looking interfaces for my bitcoin apps.

During this time, I was constantly reading bitcoin-related news stories, articles, blogs and forums, almost daily. This also provided me with a great feedback loop, as it always helped me focus on the spects of the programming language I was working with (Python and JavaScript).

Introducing ethereum

By the time Vitalik Buterin – who, by the way, was already highly respected In the bitcoin community – introduced his vision for Ethereum, I was already head over heels into Bitcoin and blockchains. I found the approach presented by Vitalik very compelling and promising, and I discovered that by learning how Ethereum works, I could enhance my own understanding of the bitcoin protocol by comparing the similarities and differences between the two.

The big leap

Meanwhile, I started to work on my Master’s degree and took few more advanced courses in algorithms and machine learning to amp up my computer skills. But I wasn’t pleased with the courses. The level at which we learned was high, no doubt about that, but none of the courses offered me any truly useful tools to work with. So I started to study by myself again.

I looked into assembly code, which helped me a lot in my understanding of the Ethereum Virtual Machine (or EVM for short). At this time, I decided to take a leap of faith and turn my passion to blockchains into a full-time career. At first, I started by creating simple videos that explained the basics of bitcoin. Then I showed some implementations for key features using Python. The initial exposure I received was very modest – but it did the job, and almost immediately I started to receive calls from professionals from a variety of industries who asked for my help and advice. Many of them wanted to use the power of bitcoin and the blockchain in their respective industries, and I was thrilled to be at the epicenter of such a great revolution in so many different and unique fields. Many of the people I worked with were genuinely looking for ways to revise their services and models to accommodate the needs of the 21st century. Many have the potential to empower millions (if not billions) of people in developing countries that might finally get direct access to many of the services we take for granted, like banking, medical insurance, pensions, legal frameworks to settle disputes, transparent government … the list goes on and on.

In the meanwhile

Blockchain architecture and computer decentralization is really not the scary thing most people think it is. The protocols are quite well documented, and once you’ve made some mental leaps, it will all start to make sense. You just need to learn it in steps. First, you need to understand what really sets blockchains apart from many other solutions. Then, using some basic coding and IT skills, you can easily familiarize yourself with many of its features and inner workings. The next step is to create some basic code that mimics a few of the more common blockchain-related features. Finally, putting it all together, you can start to create your own blockchain applications and decentralized solutions.

I’m biased. I’m in love with the vision that Bitcoin, Ethereum and blockchain architecture bring to the world. My fingers itch, and my appetite is insatiable. For years I’ve been working diligently to see this revolution spread from the computer geeks and mathematicians to the masses, where it’s really needed. So you’d better take my enthusiasm with a pinch of salt and look for the many blockchain-related projects that will now start to grow – and watch how they’re planning to literally change our world. Then you’ll feel the same way I do. I’m sure of it!

Scripts and stacks

Scripts and stacks

Personal note, Many things happened in past two months the required my full attention. I hope to resume a steady flow of posts in coming days.

Review

In the last post we’ve talked about one the biggest bitcoin misconception – The idea that transaction actually moves coins from one wallet to another. The truth is that transactions are nothing more that statements. These statements always points to a previous statement (that in turn point to an even older statement and so on), and usually these statement also specify an amount of coins that the current owner is wishing to transfer. The statement also contains a riddle, or an equation that needs to be proofed, and mostly, the key to proof this equation will require the use of the private_key that is associated with the recipient bitcoin address.

level3

Pay attention, even though Bob will be required to use his own private_key to proof that he indeed can solve this problem, the private_key still won’t be available to any one.

 

Now let’s look for a second at this transaction message. We’ve already learned how to create a bitcoin message (see this section about Version message and this one about headers). We just need to make sure that all of the fields are filled in accordance the protocol rules. Just like filling a form. You can find a complete list of the fields that needs to be filled in the bitcoin developers documentation.

 

 

Most of the fields are quite straight foreword. I might still create another post in the future with detail instructions on how to fill all the fields, but this isn’t really the topic of this post. This post deals with one of bitcoin more fascinating aspects – The riddle that Alice place in her statement. The riddle that only Bob can solve -The script.

 

(You just can’t wait to create your own transaction? you’re more than welcome to watch my videos on creating bitcoin transaction)

Scripts, what is it?

Scripts is a computer language. In more detail, it’s a set of predefined words that are agreed upon. Every node that follows the rules specified in the bitcoin protocol will know how to read, interpret and implement these words. Because bitcoin messages are basically nothing more then a string on bytes, these words are not written in plain English, rather are translated to OP_CODEs. That way, we can send our message as a string of bytes, and the receiving node will know that these bytes represent some instructions. (Important note, The receiving node will only treat this bytes as instructions only if they appear inside one of the script field.)

Here’re selected few:

Word Opcode Hex Input Output Description
OP_1ADD 139 0x8b in out  1 is added to the input.
OP_1SUB 140 0x8c in out 1 is subtracted from the input.
N/A 1-75 0x01-0x4b (special) data The next opcode bytes is data to be pushed onto the stack
OP_MIN 163 0xa3 a b out  Returns the smaller of a and b.
 OP_SHA256  168  0xa8  in  hash The input is hashed using SHA-256
OP_EQUAL 135 0x87 x1 x2 True / false Returns 1 if the inputs are exactly equal, 0 otherwise.

The original list included around 200 of these words, but currently most nodes will only support few dozes of these words. Using these few words we can create many “riddles” or state many conditions to claim the coins in our transaction message.

For example I can add the following string of bytes as my script.

0x01 0x8b 0x87 0x02 0x87

<1> <OP_1ADD> <2> <OP_EQUAL>
  1. It will take the number 1.
  2. Use the OP_CODE OP_1ADD to add 1 to it -> The output of this OP_CODE will be 2.
  3. Use the OP_CODE OP_EQUAL to make sure if the result is equal to 2. -> The output of this OP_CODE will be True.

A word of caution though, most nodes not only refuse to accept most of these OP_CODEs, they will even refuse to accept most non-standard  scripts, mainly because they want users to use standard transactions. Many nodes will not only refuse to accept a transaction with a non standard script, they’ll also refuse to transmit these transactions to other nodes.

 

Stacks

You might’ve already noticed that this script language can only be written as a list of operations. Unlike other high level languages (such as python for example) Scripts can only be used in a predefined order. This type of structure is called stack, because we’re stacking variables and data on top of each other. But not only we’re stacking them, using the stack structure also means that they’ll be processed in accordance to the order in which they were stacked.

In our previous example, the integer 1 was the first item in our stack. Then came the operation OP_1ADD which took that item as its input, processed this item by adding 1 to it, and than giving the output 2. Now the number 2 is stacked BELLOW the integer 2.

<1> <OP_1ADD> <2> <OP_EQUAL>

<2> <2> <OP_EQUAL>

The node recognize the OP_CODE <0x02> as the integer 2, so it moves on to the next item in our stack – the OP_CODE OP_EQUAL. This operation input is the two items that are directly bellow it and compere the two. If both are equal, it will return True.

<True>

 

This example code can’t be used with a standard bitcoin transaction, it’s only meant to give you a general feel on how scripts works.

You can find an example of a real transaction over here:

 

 

Give it a try with bitpy

One of bitpy newest feature is the ability to create stacks and see them in action in real time. Mind you, only few OP_CODES are currently implemented, but it might still give you a feel on how stacks works.

Example of stack using bitpy
Example of stack using bitpy

 

Simple stack architecture with python

Stack architecture can easily be implemented using arrays. After all, it’s nothing than an array of objects (variables, operations, results etc’).

In our bitpy project, under Utils/OpCodes/Codes.py I’ve created a stack class. In its most basic form, this class will only create and empty array upon initialization, followed by  2 methods only.

class Stack():

    def __init__(self):
        self.items = []

    def push(self, item):
        self.items.append(item)

    def pop(self):
        elm = self.items.pop()
        return elm
  1. push(item) append new item to the array
  2. pop(item) remove the topmost item in my array.

This should be enough to create a very basic stack class. Still, I’ve added few more methods.

class Stack():

    def __init__(self):
        self.items = []

    def isEmpty(self):
        return self.items == []

    def push(self, item):
        self.items.append(item)

    def pop(self):
        elm = self.items.pop()
        return elm

    def size(self):
        return len(self.items)

    def printStack(self):
        display = ""
        for items in self.items:
            items = str(items)
            if len(items) > 5:
                display += " " + "<"+ items[:5] + "..." + ">"
            else:
                display += " " + "<" + items + ">"
        return display

    def clear(self):
        self.items.clear()

The isEmpty method will check if our stack array is empty.

The size method will give us the size of the array.

The printStack will provide us with a visual representation of our array. Pay attention that I’ve limited the size of each item to only 5 characters so that items such as hashed messages, bitcoin addresses, keys etc’ won’t take the all screen.

The clear method will remove all items from our array.

Using this methods we can easily start implementing more advanced OP_CODE to our stack array.

def OP_DUP(self):
    elm = self.pop()
    self.items.append(elm)
    self.items.append(elm)

def OP_HASH160(self): #saved as string!
    self.push(Utils.keyUtils.keys.generate_hashed_public_key_string(self.pop()))

def OP_EQUAL(self):
    elm1 = self.pop()
    elm2 = self.pop()

    if elm1 == elm2:
        self.push(1)
    else:
        self.push(0)

def OP_VERIFY(self):
    top = self.pop()
    if top == 1:
        self.push(1)
    else:
        self.push(0)

def OP_RETURN(self, input):
    self.push(input)

 

Transaction part one – The misconceptions about the block chain

Transaction part one – The misconceptions about the block chain

Three levels of abstractification.

The are three level to understand the way the Bitcoin block chain works.

The first level
One of the most simple Bitcoin block chain abstractification

In the first level, we got those who just heard about Bitcoin for the first time. People usually thinks that the coins are just associated with the Bitcoin address. Whenever Alice sends Bob coins, she just place a statement (or transaction) in the block chain specifying that the X coins that were associated with Alice address, should now be associated with Bob address. This is of course wrong since this representation contains only 3 thing: the sender address, the receiver address and the amount of coins to be transferred. But this level of abstraction is usually the first thought most people have when first presented with the idea of the block chain.

The second level
Points to previous transacions

In the second level, people starts to understand that each transaction also contains the origin of each coin. So if Alice wants to send 5 BTC to bob, she first needs to show where she got these 5 BTC from, That is, she also needs to provide a way to point to older translations on the block chain, transactions which proves that Alice is indeed the rightful owner of these five BTC. So now what we have starts to look more like a chain. It’s no longer just a table containing the address and the current BTC balance,  rather the block chain needs to contain a list of all previous transactions. This depiction is more accurate, but still it isn’t complete – introducing the third level.

The third level
Points at previous transaction and set the condition for claiming the coins.

In the third level, people learns that a transaction is more then just a simple statement in the block chain. Transaction contains more then just information about the origin of the coins (the input), the address of the receivers (the outputs) and the amount of coins to be transferred. Rather, the transaction also contains a riddle in it, and only by solving this riddle, can the coins be claimed and transferred. So when Alice send 5 BTC to Bob, she doesn’t only pointing at the transaction from which Alice got the coins, it also solves the riddle specified in that transaction to prove that Alice is allowed to claim those coins, but that’s not the end of it, in the transaction that Alice publish on the block chain she’s also inserting another riddle, a riddle that only Bob can answer. So if Bob would like to use this coins in the future, he’ll have to solve that riddle which was provided to him by Alice. And the process goes on and on and on.

I love riddles! Or, how to solve the riddle and prove that I’m allowed to claim the coins?

A quick reminder. Signing messages and key pair.

In the previous post we’ve talked about mathematical trapdoors and hashing functions. We’ve saw how we can use these types of function so sign a message with a private key and how these signed message can be verified with the corresponding public key.

F("My name is Alice", Alice's_private_key) = signed message.

Verify(signed message, "My name is Alice", Alice's_public key) = true.

Change any component in the Verify statement, and we’ll receive false. This way we can easily prove that the message “My name is Alice” was indeed signed by Alice and that no one tempered with that message.

Pay attention! We cannot sign a message with the public key, it simply wont work! the public key can only be used to verify a message, not to sign it!

So, Alice can now publish a message in the block chain that contains the input (origin of the coins) and the output (the address of the receiver). This is the original message, public for anyone to see and check as he or she pleases. Alice can also now specify the following rules in that public message:

  1. This message is the original message.
  2. Within this message I’ve included Bob’s_public_key (In hashed format – We’ll get there in a second).
  3. Take your public key and hash it. If your hashed public key matches the hashed public key from step 2, you may move to the next step.
  4.  Take this original message, and sign it with your own private key.
    F(This message, Bob’s_private_key) = signed message.
  5. The signed message should be verified only against the public key that is specified in this message.
    Verify(signed message, This message, Bob’s_public_key) = true.
  6. If the verification indeed yield true, you may claim the coins in this transaction

 Getting Bob’s public key

When Alice specify Bob as the recipient of a transaction, she does so by inserting Bob’s hashed public key. The hashed public key (as we already saw in the post dealing with keys) can be deducted from Bob’s Bitcoin address.

The public key is hashed and then added to the final Bitcoin address.
The public key is hashed and then added to the final Bitcoin address.

So basically, knowing Bob’s Bitcoin address is equal to knowing Bob’s hashed public key. All we need to do is to convert the address back from base 58, and remember that out of the 25 bytes of the binary address, we need to omit the first byte and the last 4 bytes.

Scrips – the beginning

As we’ve already saw many times in the past, Bitcoin message is no more then a string of bytes that follow a predefined order. That means that in order to specify conditions and rules like the one Alice is using in her transaction message, we also need all parties to agree on a predefined field that will contain the conditions, and we also need to agree on a way to translate the bytes in that field into a set of rules, or instructions. For this purpose, a scripting language was created for Bitcoin, this langue allocate predefined operation to a predefined byte. For example, the byte 0x8b means “plus 1”, the byte 0x8c means “minus 1”, the byte 0xa9 means “hash the public key. First with SHA-256 and then with RIPEMD-160”, and so on. There’re many more of these operators (also called OP-codes) and these OP-codes helps us to specify the rules that needs to be fulfilled. These rules are transparent and are publicly visible on the block chain. Whenever a node verify a transaction, the nodes follows these rules and make sure that they eventually do yield a true statement.

We’ll talk more about these OP-codes and scrips in the next sections, when we’ll also see a real example of valid transaction script.

 

Keys, addresses and hashing

Keys, addresses and hashing

A key pair is one of the greatest tools that are used in Bitcoin, but it might be a little unintuitive at first. Don’t worry, you’ll get it!

There’s also a short video I made a few months ago that describes the basics of keys. It doesn’t completely corresponds to our current project, but it might provide you with another point of reference. You can watch it over here – Bitcoin python tutorial for beginners – keys and address.

 

One way function

The name “one way function” is quite self explanatory. These function are very easy to solve in one way but almost impossible to invert. Giving function f, and the input x, I can easily calculate the result y.

f(x) = y <- easy to solve

But given the result y, and the function f, It will be almost impossible to find x

f(?) = y <- almost impossible to guess.

The Bitcoin protocol define the use of some of these one way functions (SHA256, ripmed, ECDSA and murmurhash. More functions are being tested and might be used in the future).  Each one of these function has its own place in the protocol. Some functions will be used more then once and/or will be combined with another function to achieve even a grater level of security. For example, signing a message (usually a transaction message) will be done using the SHA256 function, SHA256("hello!"), finding the checksum of the message payload will be done using the SHA256 function twice SHA256(SHA256(message)).

  • Some people have hard time to accept the concept of “hard to guess”, they feel it’s too ambiguous. Well, technically an extremely powerful computer might be able to iterate through all possible results until it will find the right one (this is called brute force), but in practice, it will take a very – very long time. Trying to brute force the result of a SHA256 function on a 32 bytes message will take about 10^65 years. The age of the universe is only 1.4*10^9 year. I think it’s good enough security.
It’s easy to get the result y of function f for a giving x. But almost impossible to tell what the original input was.

 

Mathematical trapdoor and key pair

Mathematical trapdoor is a special type of one way function. The main difference is that in mathematical trapdoor we may also use few extra pieces of information called keys. Bitcoin uses the mathematical trapdoor function ECDSA or Elliptic Curve Digital Signature Algorithm, to produce two keys, or a key pair -A private key, and a public key – Both keys will always come in pairs! there cannot be a public key that matches two different private keys and vice versa!

The private key is used to solve (sign) the function f for message x. The result is the signed message y.

f(private_key, x) = y <- easy to solve

Now I have two messages. The original message x, and the signed message y. I want to prove that I’m the one who signed the original message x, that I’m the owner of the private key. But I don’t want to give my own private key. Anyone who have my private key will be able to sign in my name on other messages as well. So I’m using the public key. The public key can only be used to prove the solution of the function, but it cannot be used to sign messages

f(public_key, x) = y <- easy to prove

f(public_key, x) = null <- I can't sign message x with the public key. only with the private key

  • Pay attention that when we’re using the public key we’re just proving the equation, not solving it.

Here’s a simple numeric example I found on the wikipedia page on mathematical trapdoor:

An example of a simple mathematical trapdoor is “6895601 is the product of two prime numbers. What are those numbers?” A typical solution would be to try dividing 6895601 by several prime numbers until finding the answer. However, if one is told that 1931 is one of the numbers, one can find the answer by entering “6895601 ÷ 1931” into any calculator. This example is not a sturdy trapdoor function – modern computers can guess all of the possible answers within a second – but this sample problem could be improved by using the product of two much larger primes.

 

Let’s see an example:

Step one – create a key pair:

Create key pair using the ECDSA function and some random numbers
Create key pair using the ECDSA function and some random numbers

Step two – sign a message with the private key:

using the private key and the ECDSA algorithm
using the private key and the ECDSA algorithm

Step three – send the original message alongside the encrypted message and the public key

3 items are needed to validate the message. The public key, the original message and the encrypted message
3 items are needed to validate the message. The public key, the original message and the encrypted message

The code

In our project we’ve defined the Key class under Bitpay/Utils/KeyUtils/keys.py. This class contains all the necesery steps that are required in order to generate a private key, trnsform that private key to a public key and then create a Bitcoin address out of that public key.

step one – create (or receive) the private key

The first thing that we’re going to do is to create our private key. The private key is defined as  a random 32 bytes uint. Our class begins with a simple check. If the user initialize the Key class with an already existing private key, that private key will be saved into self.private_key. Otherwise, we’re using the urandom function in the os module to create a random 32 bytes long number.

def __init__(self, private_key=0):
    if private_key == 0:
        self.private_key = os.urandom(32)
        self.printable_pk = str(binascii.hexlify(self.private_key), "ascii")
    else:
        self.printable_pk = private_key
        self.private_key = binascii.unhexlify(private_key.encode('ascii'))

You might’ve noticed that we’ve also created a printable_pk variable. This variable will store the private key in hexadecimals. This way it is easier to store, copy and/or print the private key.

 

Step two – Use the private key to initialize the signing function

After we got our private key it’s time to use it initialize our ECDSA function. This step is similar to declaring our function f with the private key pr_k.

self.sk = f(pr_k, )

We’re defining the variable self.sk (for Signing Key) and use SigningKey.from_string from the ECDSA module with two arguments, the first one is our self.private key, and the second one is the curve (We haven’t talked about the curve yet, But it represent the mathematical part of our function. This is too advance mathematics so we won’t go into it in this project. But for now we just need to know that the Bitcoin protocol requires us to use the ECDSA function with the mathematical curve SECP256k1)

self.sk = ecdsa.SigningKey.from_string(self.private_key, curve = ecdsa.SECP256k1)

 

Step three – Use the initialized function (self.sk) to get the public key

Now that we got our signing key, we can use it in order to create our public key.

self.vk = self.sk.verifying_key

We’re defining a new variable called self.vk which will hold the verifying key, or the public key that can be sent alongside the signed message and the original message. This key will be used to verify that the message was indeed signed by the owner of that public key. And since every public key matches only one specific private key, it also proves that the one who signed the message also possess the corresponding private key.

 

Step four – Formatting the public key.

The variable self.vk holds the public key that will be used to verify our signed messages. But the Bitcoin protocol requires that we’ll represent this public key in couple of different formats.

The following chart from the Bitcoin wiki site shows the way the public key should be formatted:

Converting the public key to Bitcoin address
Converting the public key to Bitcoin address

 

The first line is the real public key, or in our case the verification key self.vk

this is the real public key - but we can't send it like this. We need to do dome formatting
this is the real public key – but we can’t send it like this. We need to do dome formatting

 

The second line tells us that we need to inser the byte 0x04 at the beginning of our public key

self.public_key =  b"04" + binascii.hexlify(self.vk.to_string())

We’re using the function to_string in order to display the variable self.vk  as a string. Then we convert it to hexadecimals so it will be easier to append the byte 0x04.

This is the public key in Bitcoin terminology. Usually, When looking for the public key in signed transactions, that's what it will look like
This is the public key in Bitcoin terminology. Usually, When looking for the public key in signed transactions, that’s what it will look like

 

The third line tells us to hash the public key twice. once using the SHA256 function, and then again using the ripemd160 function.

ripemd160 = hashlib.new('ripemd160') # <-initializing the ripemd160 function 
ripemd160.update(hashlib.sha256(binascii.unhexlify(self.public_key)).digest())
First hashing the public key using the SHA256 function. Then the result is hashed with the ripemd160 function
First hashing the public key using the SHA256 function. Then the result is hashed with the ripemd160 function

The forth line tells us to add another byte at the beginning of the hashed key.

We're working with the main network so we'll add the byte 0x00
We’re working with the main network so we’ll add the byte 0x00

This is the network ID byte which is used to prevent us from using keys and addresses that were generated in the test network, in the main network (and vice versa). In our example we’re using the main network, so the byte we’ll add will be 0x00.

self.hashed_public_key = b”00″ + binascii.hexlify(ripemd160.digest())

In Bitcoin terminology, the result is the hashed public key. This format is used mostly when creating a transactions.

 

The fifth (and sixth) line tells us to take our hashed public key and hash it again, twice, using the SHA256 function. The first 4 bytes of the result will be the checksum.

self.checksum = binascii.hexlify(hashlib.sha256(hashlib.sha256(binascii.unhexlify(self.hashed_public_key)).digest()).digest()[:4])
The checksum is the first 4 bytes
The checksum is the first 4 bytes

 

The seventh line creates the Bitcoin address in its binary form by appending the hashed public key with the checksum. This is a valid Bitcoin address, but it still need to go through one more process before it can be used with most Bitcoin wallets.

self.binary_addr = binascii.unhexlify(self.hashed_public_key + self.checksum)

 

The last line Finally we’ve reached the end point. There’s only one more thing we need to do before we can get the standard Bitcoin address and that is to convert the binary code of the address into a base58 string. The idea behind this conversion is quite simple. In order to reduce human errors, it was decided that some characters will be omitted from the standard Bitcoin address. characters like capital O, the number 0, lower case l and upper case I, as well as many more characters were omitted.

The final address represented in base 58.
The final address represented in base 58.
self.addr = base58.b58encode(self.binary_addr)
  • You might need to install the base58 module using the command pip install base58.

 

User interface

We’ve also added a tab to our graphical user interface which might help. You can use it to see the public key, hashed public key and Bitcoin address or any given private address.

The user interface for the keys can be found in the second tab
The user interface for the keys can be found in the second tab
Connection part three – Receiving messages

Connection part three – Receiving messages

In the previous posts, all that we’ve done was to construct and send messages to another node on the network. In this post, we’ll see what happens to incoming messages.

First stop – The ReceiverManager:

class ReceiverManager(Thread):
    def __init__(self, sock):
        Thread.__init__(self)
        self.sendingQueue = Utils.globals.sendingQueue
        self.sock = sock
        self.ping = ""

        self.outfile = open("data_received_from_node.txt", 'w')

    def run(self):
        while True:
            try:

                # get only the header's message
                header = self.sock.recv(24)

                if len(header) <= 0:
                    raise Exception("Node disconnected (received 0bit length message)")

                headerStream = BytesIO(header)
                parsedHeader = HeaderParser(headerStream)

                # get the payload
                payload = self.recvall(parsedHeader.payload_size)
                payloadStream = BytesIO(payload)

                self.manager(parsedHeader, payloadStream)

            except Exception as e:
                print(e)
                break

        print("Exit receiver Thread")

The receivermanager always runs in the background, checking our Thread for any incoming packets. Once it receives a packet, it will immediately cut its first 24 bytes.

header = self.sock.recv(24)

The first 24 bytes are the header. If you remember from this post, every Bitcoin message will starts with header, and the header is always exactly 24 bytes long.

 

The first 24 bytes are the header. The rest is the payload
The first 24 bytes are the header. The rest is the payload.

This header is now parsed as a string of bytes and passed to the HeaderParser class in Bitpy/Network/HeaderParser.py

headerStream = BytesIO(header)
parsedHeader = HeaderParser(headerStream)

 

Second stop – The HeaderParser class:

The HeaderParser class takes the first 24 bytes as a long string of bytes, and then it reads them in the same order that we’ve seen before.

Size (Bytes) Name Data type Description
4 Start string char[4]  The network identifier
12 Command name char[12]  The name of the command.
4 Payload size uint32 Len(payload)
4 Checksum char[4]  SHA256(SHA256(payload))[:4]

First 4 bytes for the Start string (or Magic number), another 12 bytes for Command name, the next 4 bytes are the Payload size and the last 4 bytes are the checksum.

4 bytes for starting string. 12 for command name. 4 for payload size and 4 for checksum
4 bytes for starting string. 12 for command name. 4 for payload size and 4 for checksum
class HeaderParser:
    def __init__(self, header):  # Packets is a stream

        self.magic = read_hexa(header.read(4))
        self.command = header.read(12)
        self.payload_size = read_uint32(header.read(4))
        self.checksum = read_hexa(header.read(4))

        self.header_size = 4 + 12 + 4 + 4

    def to_string(self):
        display = "\n-------------HEADER-------------"
        display += "\nMagic:\t %s" % self.magic
        display += "\nCommand name	:\t %s" % self.command
        display += "\nPayload size	:\t %s" % self.payload_size
        display += "\nChecksum	:\t\t %s" % self.checksum
        display += "\nheader Size:\t\t %s" % self.header_size
        display += "\n"
        return display

We’ve also defined the to_string function which basically makes it easier to print a human readable version of the message header.

You might’ve noticed that currently our code just accept the checksum field from the received message without checking it. This is of course a security flaw in our code. The checksum filed is there to help us verify the authenticity of the message. That is one of the ways we can make sure that no one tempered or changed the message on its way from the sender node to our node. But for the time being we’ll assume that the message is indeed authentic and we’ll accept the checksum as is.

 

Third stop – Back to the ReceiverManager:

Now that we have our header, it’s time to get the payload. The size of the payload was defined in the header of the message. We need to cut that amount of bytes from our incoming packets, just as we cut the first 24 bytes of the header. There’s however one extra step in our code. Instead of using the built in sock.recv function (as we did for the header) we’ve decided to implement our own recevall function. The rational was that since we have no way to predetermine the size of the payload, and since the built in sock.recv can’t handle large packets of unknown size, it would be wiser to break the payload into smaller parts and append them together. This has nothing to do with the Bitcoin protocol, it’s only our way to make sure that the code will properly handle large messages.

def recvall(self, length):
    parts = []

    while length > 0:
        part = self.sock.recv(length)
        if not part:
            raise EOFError('socket closed with %d bytes left in this part'.format(length))

        length -= len(part)
        parts.append(part)

    return b''.join(parts)

So now, after we’ve cut the required amount of bytes that represents the payload of our message, and we have both our header (which was already parsed) and our payload (yet to be parsed), we’ll pass them both to the receivermanager manager function.

 

Forth stop – Manager:

    
def manager(self, parsedHeader, payloadStream):

    command = parsedHeader.command.decode("utf-8")
    message = {"timestamp": time.time(), "command": command, "header": parsedHeader.to_string(), "payload": ""}


    if command.startswith('ping'):
        ping = Ping.DecodePing(payloadStream)

        pong = Pong.EncodePong(ping.nonce)
        packet = PacketCreator(pong)
        self.sendingQueue.put(packet.forge_packet())

        message["payload"] = str(ping.nonce)
        self.display(message)

    elif command.startswith('inv'):
        inv = Inv.DecodeInv(payloadStream)
        message["payload"] = inv.get_decoded_info()
        self.display(message)

    elif command.startswith('addr'):
        addr = Addr.DecodeAddr(payloadStream)
        message["payload"] = addr.get_decoded_info()
        self.display(message)

    elif command.startswith('pong'):
        pong = Pong.DecodePong(payloadStream)
        message["payload"] = pong.get_decoded_info()
        self.display(message)

    elif command.startswith('version'):
        version = Version.DecodeVersion(payloadStream)
        message["payload"] = version.get_decoded_info()
        self.display(message)

The manager function does a very simple thing. It checks the command of the message (the command is part of the header) and then it sends the message payload to be parsed by the corresponding functions. For example. If the manager sees that the command is «pong», it will use the decodepong method in Bitpay/Packets/control_messages/pong.py to extract the desire fields out of it. (You can read more about «pong», «ping» and «verack» messages in this post.).

 

Divergence

We have our pared message, both its header and payload. And now we need to decide what to do with them. For some messages this might be the end of the line. There’s nothing more we can do with them. Some might require us to act. «ping» message should be answered by a «pong» message, transactions should be checked and relayed (We’ll talk about transactions in later posts), «version» messages should be acknowledged by sending back a «verack» message.

A major part of learning the Bitcoin protocol is learning how each and every message should be dealt with. Which fields of information it contains and what is the meaning of this information. We’ve already talked about some of the messages in previous posts  (see here for «ping», «pong» and «verack» messages, and here for «version» message.) and as our project will have more features implemented, so we’ll discuss other type of messages and how to deal with them.