mkdir /var/lib/docker: read-only file system

Got this error trying to launch a docker service. After some searching came across this link. Indeed the docker from snap is broken. Once I installed using official steps, the error wen’t away. Please don’t install docker using `sudo snap install docker`.

Posted in Software | Leave a comment

“error”:”invalid_grant”,”error_description”:”AADSTS700082: The refresh token has expired

Sometimes you will get this error when trying to run az CLI. My first attempt to fix it was to refresh the token using az account get-access-token but when I ran it, it gave this:

Get Token request returned http error: 400 and server response: {“error”:”invalid_grant”,”error_description”:”AADSTS70002: Error validating credentials. AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2018-10-10T22:50:37.2119047Z and was inactive for 90.00:00:00.\r\nTrace ID: 2d6dfd99-0307-4f8d-a5d7-f259b6f51000\r\nCorrelation ID: 6876283d-f5a2-4dfc-9549-882b750f60bb\r\nTimestamp: 2019-01-28 21:00:43Z”,”error_codes”:[70002,700082],”timestamp”:”2019-01-28 21:00:43Z”,”trace_id”:”2d6dfd99-0307-4f8d-a5d7-f259b6f51000″,”correlation_id”:”6876283d-f5a2-4dfc-9549-882b750f60bb”}

The solution is to run az login to refresh the token.

Posted in Software | Leave a comment

Docker Swarm Mode Ports 2377, 7946, 4789

The docker docs say that for swarm mode to work:

Open protocols and ports between the hosts

The following ports must be available. On some systems, these ports are open by default.

  • TCP port 2377 for cluster management communications
  • TCP and UDP port 7946 for communication among nodes
  • UDP port 4789 for overlay network traffic

If you plan on creating an overlay network with encryption (--opt encrypted), you also need to ensure ip protocol 50 (ESP) traffic is allowed

Here is how one can test if these ports are indeed open. Use the nc command like follows:

On swarm manager:

siddjain@goldenrwr-ca0:~$ nc -zv localhost 2377

Connection to localhost 2377 port [tcp/*] succeeded!

siddjain@goldenrwr-ca0:~$ nc -zv localhost 7946

Connection to localhost 7946 port [tcp/*] succeeded!

siddjain@goldenrwr-ca0:~$ nc -uzv localhost 4789

Connection to localhost 4789 port [udp/*] succeeded!

siddjain@goldenrwr-ca0:~$ nc -uzv localhost 7946

Connection to localhost 7946 port [udp/*] succeeded!

On swarm worker note that port 2377 is refused:

siddjain@goldenrwr-peer0:~$ nc -zv localhost 2377

nc: connect to localhost port 2377 (tcp) failed: Connection refused

siddjain@goldenrwr-peer0:~$ nc -zv localhost 7946

Connection to localhost 7946 port [tcp/*] succeeded!

siddjain@goldenrwr-peer0:~$ nc -uzv localhost 4789

Connection to localhost 4789 port [udp/*] succeeded!

siddjain@goldenrwr-peer0:~$ nc -uzv localhost 7946

Connection to localhost 7946 port [udp/*] succeeded!

Posted in Software | Leave a comment

Running Custom Build of Hyperledger Fabric Orderer

If you want to run it inside a docker container:

If you want to run a custom build of the orderer, make sure you set the FABRIC_CFG_PATH and have an orderer.yaml inside FABRIC_CFG_PATH. The docker container will have it setup automatically for you. Copy the orderer.yaml from container to local machine using docker cp.

Install fabric from github under your $GOPATH/src/

To build orderer simply run go build from the orderer folder. The orderer will require a few variables to be set before you can run it like CHANNEL_TX_FILE=./channel.tx GENESIS_BLOCK_FILE=./genesis.block ORDERER_GENERAL_GENESISFILE=./genesis.block ORDERER_GENERAL_LOCALMSPID=ordMSP ORDERER_GENERAL_LOCALMSPDIR=./msp.

One handy command is docker inspect <image-name>. If we run it on fabric-ca-orderer, among other things it shows

 “Env”: [




            “Cmd”: [



Cmd contains the command the container will execute on starting. There is another field called Entrypoint which does the same thing. When both an ENTRYPOINT and CMD are specified, the CMD string(s) will be appended to the ENTRYPOINT in order to generate the container’s command string. More details at

Posted in Software | Leave a comment

VS Code Setup

Getting Bash IDE to work in VS Code:

Installing bash-language-server in non global mode:

Open VS Code settings by clicking on gear icon at bottom left of VS Code

To Install/Update Go Tools View -> Command Palette (Cmd+Shift+P)

View -> Command Palette (Cmd+Shift+P)

To open a file quickly, Cmd+P

Posted in Software | Leave a comment

Ethereum vs. Hyperledger: The Epilogue

I was very excited to work with Hyperledger when I first did a comparison between Ethereum vs. Hyperledger [1]. The ability to write chaincode in javascript was especially appealing to me. However after more than a month of spending time in the trenches and doing Hyperledger Fabric development, my opinions changed. If I were to revisit Part I of Ethereum vs. Hyperledger [1], I would add one more row to the table as follows:

Developer ExperienceF

I haven’t done any programming with Ethereum so don’t know how developer friendly it is but can safely say that Hyperledger scores an F on the developer experience. Lets try to break down the developer experience on following competencies:

  1. Works without any issues
  2. SDK provides good, exhaustive and bug-free code samples showing how to write code and exercise functionality
  3. Comprehensive and accurate documentation
  4. Tech Support: When there is an issue, there is a helpful community to provide support

Rating Scale.

A – excellent. Scores +1 on all dimensions above
B – Good. Better than other alternatives out there
C – Fair. comparable to other alternative platforms
D – Poor. there exist better alternatives out there
F – Fail. -1 on all of the above dimensions

Hyperledger proudly scores an F

  • Broken out of the box: e.g., Its like you ordered something from amazon and its broken out of the box. Looks like HL team spend no time testing the code works on a mac
  • Buggy Samples: e.g., Another example:
  • Samples of limited use and help: e.g.,
    • everywhere in the samples they keep on using cryptogen and then write that cryptogen should not be used in production.
    • In all the samples, the private crypto keys are blatantly exposed [example] and there is no sample showing how to protect them in a prod environment.
    • All the samples create a network in which all the nodes are running on the same computer. There is no sample showing how to create a real-world network spanning multiple computers
    • the list goes on. Basically there is no sample showing how to write a production quality app
  • Incorrect documentation: e.g., instructions on say that Node.js v6.9.0 – 6.10.0 ( Node v7+ is not supported ) is required whereas the instructions elsewhere say that Node.js v8.4.0 or higher. Even the HL Fabric prerequisites say that If you will be developing applications for Hyperledger Fabric leveraging the Hyperledger Fabric SDK for Node.js, you will need to have version 8.9.x of Node.js installed. I kept a screenshot of the incorrect documentation here in case it gets fixed later. This e.g., caused me lot of trouble and wasted time. As I found out thae hard way, in reality Node.js v6.9.0-6.10.0 will land you in trouble. This is because Promises were introduced only in version 8 of Node.
  • Missing Documentation: Hyperledger Fabric relies on a dozen configuration files. Some of them are listed below:
    • configtx.yaml
    • orderer.yaml
    • fabric-ca-server-config.yaml
    • fabric-ca-client-config.yaml
    • core.yaml
    • peer.yaml

nowhere can one find explanation of all the fields in these yaml files. More examples showing the sad state of documentation [1, 2]

Some things that are unrelated to developer experience but worth mentioning (identifying open gaps):

  • No support for kubernetes [ref]. Running HL Fabric on Kubernetes requires that you provide privileged access in order to run the chaincode. There are also some other issues as well. As a result, running HLF on kubernetes and thus OpenShift, is not recommended for production.
  • I am also not sure if Hyperledger Fabric really comes with a true consensus protocol [1]
  • And it seems to come with a some security loopholes [ref]. Essentially in order to run fabric, you have to allow peer nodes access to the the docker daemon which they need in order to spin up a new container to run the chaincode; but giving this access is risky as illustrated in [ref]. That is why there is no support for kubernetes.
Posted in Software | Leave a comment

Ethereum vs. Hyperledger – Which is better? Part I

Built for enterprise





Ability to write chaincode in familiar languages



Transactions can be non-deterministic



Support for private data


Enterprise Footprint


Cryptocurrency not needed


Membership Service Provider


  • Built For Enterprise:
“most enterprise apps would get tilted towards Fabric, whereas Ethereum would continue to be a hotbed for dApps that are more B2C”
Ethereum has the EVM running the Smart Contracts for applications that are attributed to being decentralized and are for mass consumption.
On the other hand, Hyperledger leverages blockchain technology for business.

  • Scalability:
  1. Ethereum uses a PoW consensus which is expensive whereas HL uses Apache Kafka to order the transactions and a customizable endorsement policy to drive consensus. Fabric claims its consensus protocol can be changed and customized.
  2. Ethereum uses order-execute architecture which requires all peers to execute every transaction whereas HL uses execute-order-validate architecture
  3. Ethereum can process 25 transactions per sec [1] whereas HL Fabric can process 3500 TPS [2]
  4. The experimental results, based on varying number of transactions, show that Hyperledger Fabric consistently outperforms Ethereum across all evaluation metrics which are execution time, latency and throughput. [3]. Also see [4] for more independent tests confirming the same
  • Ability to write chaincode in familiar languages: Ethereum smart contracts have to be written in Solidity language. HL Fabric chaincode can be written in Go, NodeJS or Java
  • Transactions can be non-deterministic: this is the reason why Ethereum does not support writing smart contract in a general purpose programming language
  • Support for private data: HL provides support for private data using channels and side-db. In Ethereum there are no means to issue a private transaction between members [3].
  • Enterprise Footprint: This is the most important factor that should be used to guide which platform to choose. Many big and reputed enterprises have deployed HL Fabric based blockchain solutions e.g.:, AID:Tech, MedicalChain. I am not aware of any enterprise blockchain based on Ethereum (although I have not done exhaustive research here; also see this). JPMorgan has taken a fork of Ethereum and developed a new offering known as Quorum. there are enterprise apps built on Quorum. But Quorum is not Ethereum.
  • Cryptocurrency not needed: Ethereum requires use of a cryptocurrency known as Ether. HL Fabric has no concept of a cryptocurrency as it is N/A for enterprise apps.
  • Membership Service Provider: A MSP is what enables creation of a permissioned blockchain. Ethereum does not some with any MSP.

It is possible that Ethereum team will work to address the issues above so that its used in enterprise (see this). But in the meantime there is no reason not to use a platform that was specifically built for business.

Posted in Software | Tagged | Leave a comment