Notes on Programming Hyperledger Fabric

I just finished writing a book:
https://www.amazon.com/dp/0578802228

In this post I am making some notes and fun facts about the book for remembrance.

Original TOC

Final TOC

Blockchain Primer Introducing Hyperledger Fabric A proof of concept app Setting up your infrastructure Securing communications with OpenSSL Establishing the user registry with LDAP Generating certificates with Fabric CA Membership services and policies Transaction ordering The Peers Channels Implementing business logic using Chaincode Fabric Node.js SDK Advanced Topics Appendix A: Installation and setup

Blockchain Primer A Closer Look at Fabric Writing Your First Smart Contract Generating Identities Required to Bootstrap a Network Provisioning a Three Org Test or Development Network Debugging Fabric Provisioning Channel and Installing Chaincode Performing Transactions on the Network Updating Channel Configuration Dockerizing the Network Securing Communications with TLS Handling Data Privacy Running Fabric CA Server Using Fabric CA Client to obtain X.509 Certificates Authenticating Users using LDAP Bonus Chapter: The Bitcoin Appendix A: Installing Software and Prerequisites Appendix B: Summary of Commands

And here is a word cloud – list of most commonly occurring words in the book curated to remove common words like like, when, which, user, using, then, what etc.

$ python counter.py full.adoc
[('Fabric', 447), ('peer', 446), ('chaincode', 330), ('certificate', 314), ('Docker', 290), ('orderer', 287), ('client', 285), ('block', 250), ('server', 237), ('container', 224)]

The full manuscript itself is about 1MB in size containing about 150,000 words

$ ls -al full.adoc
-rw-r--r--  1 sjain68  staff  1033787 Dec 17 10:26 full.adoc
$ wc -w full.adoc
  151713 full.adoc

VS Code + AsciiDoc

Today I learned that VS Code has a pretty good extension for AsciiDoc:

https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode

It can even export to PDF using wkhtmltopdf. There is also the option to have it use the Ruby based asciidoctor-pdf to do the conversion.

How to Use

The extension activates automatically when opening an AsciiDoc file (.adoc, .ad, .asc, .asciidoc).

Preview

To show the preview you can use the same commands as the Markdown extension:

• Toggle Preview – ctrl+shift+v (Mac: cmd+shift+v)
• Open Preview to the Side – ctrl+k v (Mac: cmd+k v)

The preview refreshes automatically, but it can also be forced with the AsciiDoc: Refresh Preview command.

The preview supports setting attributes through the asciidoc.preview.attributes option.

By default the preview style follows the VSCode theme (workbench.colorTheme). To use Asciidoctor’s style set option asciidoc.preview.useEditorStyle to false. It is also possible to set your own preview stylesheet with the asciidoc.preview.style option.

(See more details under Options)

Export as PDF

The extension provides a quick command to export your AsciiDoc file as PDF.

• Open the command palette – ctrl+shift+p or F1 (Mac: cmd+shift+p)
• Select AsciiDoc: Export document as PDF
• Choose the folder and filename for the generated PDF

By default a separate binary is downloaded and used to render the document in PDF format. To use Asciidoctor PDF set option asciidoc.use_asciidoctorpdf to true.
(See more details under Options)

To change the settings first open the extensions tab. then select Asciidoc extension. click on gear icon as shown below

also see https://stackoverflow.com/questions/50477523/how-to-convert-asciidoc-to-pdf/50709725

Using AsciidocFX

AsciidocFX is one dangerous program. I had it open like this. It had my root folder open in left pane and I was used to Atom where one can remove the folder by right clicking the folder and then remove

I did same thing in AsciidocFX and clicked on Delete. And what does it do? It actually deletes (rm) the folder!

Next I tried to find where can I set the Workdir. There is a settings tab on the right

but it doesn’t show Workdir setting anywhere. You can find all the settings under

$ ls /Users/siddjain/Applications/AsciidocFX/conf
ace_doctypes.json		asciidoctor_html.json		cheatsheet			editor_config.json		public				spellcheck_config.json		themes
ace_themes.txt			asciidoctor_preview.json	docbook				extension_config.json		slide				stored_directories.json
asciidoctor_docbook.json	booksample			docbook-config			location_config.json		spellcheck			terminal_config.json

But here also I did not find Workdir. It turns out you simply double-click on the folder in the file explorer tab to change the Workdir. The left pane is to be used as a file explorer.

AsciidocFX uses a Java converter to convert adoc to PDF. This is very different from asciidoctor-pdf which uses a Ruby converter. asciidoctor-pdf converter is more mature than AsciidocFX. The future of Asciidoctor is asciidoctor-web-pdf which will convert adoc to HTML and then run a headless Chrome to save the HTML as PDF. The project is in early stage.

AsciidocFX is okay for one-pagers etc. For anything serious you should use asciidoctor-pdf. It is faster. I like using asciidoctor-pdf and Adobe Brackets to edit adoc files. Brackets comes with a asciidoc previewer and Alice offline spell checker.

Useful Docker commands

Sort images by size

docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}" | sort -k 2 -h

List dangling volumes

docker volume ls -f dangling=true --format '{{.Name}}'

Cleaning up Docker resources: https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes#a-docker-cheat-sheet

Cross-referencing content of another chapter in AsciiDoctorPDF

It is possible to cross-reference chapters in Asciidoc. The way I do it is as follows. First of all I have a top level index.adoc which is the root file that does nothing but include the various chapters one after another. I run the asciidoctor-pdf program on index.adoc. The include directive literally includes the content of another file. What this means is that if you have a 10 line index.adoc which is just including 10 chapters, the end result is as if you had manually copied and pasted the content of those 10 chapters into index.adoc and then run asciidoctor-pdf on top of it.

So in the end its as if asciidoctor-pdf was run on a single file. The normal rules of cross-referncing apply. In documentation you will read that to cross-reference a section from another chapter you have to prefix with filename of the chapter like so <<chap10.adoc#results>>. However you precisely don’t have to do this – at least if you are using the setup described above. Everything ends up as one single file you can can cross-reference content of another chapter like <<results>>. Indeed if you have two chapters and you give the same ID to two sections then there is a collision and conflict at time of cross-referencing. asciidoctor-pdf will issue an error.

Before beginning each chapter I have following preamble stored in a common-settings.adoc:

:figure-number: 0
:listing-number: 0
:table-number: 0
:figure-caption: Figure {counter:chapter}.
:listing-caption: Listing {chapter}.
:table-caption: Table {chapter}.

The figure-number , listing-number and table-number seem to be in-built counters whereas chapter is a counter I define in my book’s preamble.

One tip: With this setup, AsciiDoctorPDF runs into trouble cross-referencing listings and tables of other chapters (cross-referencing within chapter works as expected). Rather than hacking your way around it – which is very difficult – cross-reference the relevant section.

Getting Math to work in AsciiDoctorPDF

I had quite a lot of challenges getting math to render in AsciiDoctorPDF. The documentation relating to this is also confusing. It turns out that for converting asciidoc to PDF which is what I am interested in (i.e., output format = PDF) AsciiDoctorPDF uses asciidoctor-mathematical library. The MathJax library does not apply when output format = PDF. I wasn’t able to install asciidoctor-mathematical successfully on my Mac but the Docker image docker-asciidoctor provided a solution until I ran into gsub!': asciidoctor: FAILED to load AsciiDoc document - can't modify frozen String (FrozenError) error. This was fixed in asciidoctor-mathematical 0.3.1 (see this) and related changes continue to be made (see this). Using stem:asciimath did not work well for me. Its better to set

:stem: latexmath

in your adoc file. There are many other issues I ran into such as equation numbers appearing to the left instead of right. They appear to have been fixed with successive releases.

One issue that remains unresolved has to do with :imagesdir: This attribute controls the directory where AsciiDoctorPDF expects to find images. What happens is that asciidoctor-mathematical does not know anything about :imagesdir: The way asciidoctor-mathematical works is that it renders the math as a .png and then the parent asciidoctor-pdf program inserts the png into the document like any other png. That is my understanding. The problem is that if you have set :imagesdir: to say images the parent program looks for the png under that directory but asciidoctor-mathematical outputs the pngs into the working directory. The way I have been working around this is a two-step process in which I run asciidoctor-pdf twice. In the first pass I let asciidoctor-mathematical generate the images (pngs) and before running asciidoctor-pdf again, I copy over the images to the correct :imagesdir: directory. That’s what asciidoctor-mathematical should be doing in the first place.

An example equation looks like this:

[latexmath#equation1, reftext='Equation {chapter}.{counter:equation}']
++++
\begin{equation}
sig = enc(\phi(M), s)
\end{equation}
++++

This assumes you have defined two counters in your adoc like so:

:chapter: 3
:equation: 0

Now you will refer to this equation as <<equation1>> in your adoc. And the reftext controls what text will be inserted in place of <<equation1>> . The counter keyword increments the counter so that you get what you want. The attributes above have nothing to do with equation numbering – the number that appears next to an equation. To get that there is one additional attribute you should define in the adoc:

:eqnums: all

This still does not fix one thing which is the format you want to use for equation numbers. The reftext only controls the string used to refer the equation. It seems we have no control over the format of equation numbers. E.g., I wanted equations to be numbered {chapter}.{equation-number} and looks like it is not possible to insert the {chapter}. prefix before the equation number. TL;DR is that if your document is math heavy AsciiDoctorPDF might not be the right choice in which to write it.

It look a long time to discover all the nuances as the documentation is not very good. Hopefully it helps someone else.

AsciiDoctor: Inspecting Environment settings of Ruby

Within the docker-asciidoctor container you can see Ruby settings like so

bash-5.0# gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.2
  - RUBY VERSION: 2.7.1 (2020-03-31 patchlevel 83) [x86_64-linux-musl]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/2.7.0
  - USER INSTALLATION DIRECTORY: /root/.gem/ruby/2.7.0
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /root/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
    - ruby
  - GEM PATHS:
     - /usr/lib/ruby/gems/2.7.0
     - /root/.gem/ruby/2.7.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin

How to get rid off annoying border around SVG images in AsciiDoctorPDF

AsciiDoctorPDF inserts an annoying black border around SVG images generated by graphviz. E.g., given this SVG image which was generated by graphviz, if you open it in chrome you can see it has no black border as shown below:

but if the same image is included in an asciidoc document like so:

image::test.svg[]

and then converted to PDF using AsciiDoctorPDF the result looks like this:

Upon investigation the reason for this has to do with this line in the SVG:

<polygon fill="white" stroke="transparent" points="-4,4 -4,-403 220,-403 220,4 -4,4"/>

Chrome and other programs correctly interpret transparent to mean a transparent border but AsciiDoctorPDF – or rather the Prawn library it uses under the covers – does not. The fix for this is simple. I use the asciidoctor/docker-asciidoctor:1.2.0 image. There is a file /usr/lib/ruby/gems/2.7.0/gems/prawn-svg-0.30.0/lib/prawn/svg/color.rb in the container. Open this file in a text editor and just add following line to it:

'transparent' => 'ffffff',

under HTML_COLORS dictionary. Voila! That’s it! Problem solved! You are welcome. This will render transparent stroke as white which works for me as my page background is white. Change it if needed to adapt to colour of your page background.

Ideally graphviz should not be generating any polygons with transparent stroke as transparent is not part of the SVG 1.1 spec, but being the sucker it does. So the root cause of the problem is with GraphViz.

Validating PDFs

Sometimes you need to validate a PDF file. Here are step by step instructions on how to do so. Prerequisites: You need to have Java installed on your machine.

1. Download preflight-app-2.0.21.jar from https://pdfbox.apache.org/download.cgi
2. Run it passing the path to your PDF file like so:
   

$ java -cp preflight-app-2.0.21.jar org.apache.pdfbox.preflight.Validator_A1b test.pdf

References:

• https://github.com/apache/pdfbox/blob/2.0.21/preflight/src/main/java/org/apache/pdfbox/preflight/Validator_A1b.java
• https://github.com/apache/pdfbox/blob/2.0.21/preflight-app/pom.xml
• https://pdfbox.apache.org/1.8/cookbook/pdfavalidation.html
• https://www.pdfa.org/wp-content/until2016_uploads/2011/08/PDFA-in-a-Nutshell_1b.pdf

Extracting images from a PDF

Sometimes you need to extract images inside a PDF. Here are step by step instructions on how to do that. Prerequisites: You need to have Java installed on your machine.

1. Download pdfbox-app-1.8.16.jar from https://pdfbox.apache.org/download.cgi. WARNING: version 2.0.21 will not work.
2. Run the jar file passing it path to your PDF file like so:

java -classpath pdfbox-app-1.8.16.jar org.apache.pdfbox.ExtractImages test.pdf

That’s it. This will extract all images it can find inside the PDF and store them as test-1, test-2 and so on in the same directory as your PDF. Reference: https://pdfbox.apache.org/docs/1.8.11/javadocs/org/apache/pdfbox/ExtractImages.html