A geek stranded on Martha’s Vineyard

How to Build GnuPG 2 from Source on OS X

September 01, 2018

I’m a big fan of homebrew, but when it comes to encryption, I get paranoid and prefer to build the software myself.

On the other hand, building Unix software from source is actually pretty simple. If you’ve never done it this is a good opportunity to try. GnuPG requires several packages so you’ll be able to exercise several times.

Versions

These instructions were tested with gnupg-2.2.9 on macOS High Sierra 10.13.6, but they should be useful for other versions as well.

List of Packages

GnuPG requires installing several packages, the download page has all of them, but this is more than you need. It’s better to follow the list that comes in the README instead, so let’s start downloading and extracting GnuPG.

The location of the tar file comes from the download link on the download page

cd /usr/local/src # prefered location for sources
curl -o gnupg-2.2.9.tar.bz2 \
  https://gnupg.org/ftp/gcrypt/gnupg/gnupg-2.2.9.tar.bz2

Before uncompressing the tarball you need to verify that this is the original and umodified version of the source. If you have an existing installation of gpg you can verify its signature. Otherwise you have to compare the file checksum, which is what I did. The list of checksums is on the integrity page.

openssl sha1 gnupg-2.2.9.tar.bz2

Once you have verified that the checksum of the output of this command matches the checksum listed on integrity page you can uncompress it.

tar -xvzf gnupg-2.2.9.tar.bz2
cd gnupg-2.2.9
cat README

The BUILD INSTRUCTIONS section of the README lists the required packages. It also has all the instructions you need.

Building from Source

The steps to build from source are the same for every package, for example, for npth they would be:

cd /usr/local/src
curl -o npth-1.6.tar.bz2 \
  https://gnupg.org/ftp/gcrypt/npth/npth-1.6.tar.bz2
openssl sha1 npth-1.6.tar.bz2
# compare the output against gnupg.org/download/integrity_check.html
tar -xvzf npth-1.6.tar.bz2
cd npth-1.6
./configure # no options except for pinentry (see below)
make
make check
make install

The ./configure script gathers information about your environment: the command to invoke the C compiler, location of libraries, etc. It verifies that all dependencies are met. It also generates the Makefile and defines some constants.

Calling make will compile all sources using the Makefile generated by the previous step.

The make check is not required but a good idea to detect any issues. In the case of libgcrypt it also runs some benchmarks, but for me they run for too long and I ended up canceling it.

Sometimes make or make check will issue errors. I didn’t run into any in this case, but in the past, for older versions of GPG, I did get one, when the location of files was different; in other instance some constant definitions were missing.

make install will copy executables and libraries and create symbolic links in directories where they can be accessed. Sometimes you might need extra permissions to finish this step and have to execute using sudo. It didn’t happen to me.

Pinentry Options

For all the packages I used the defaults, i.e. didn’t provide any options to ./configure This was the exception. Pinentry is used to enter passwords/passphrases, since I’m a terminal-type person I like to use tty:

./configure \
  --disable-pinentry-qt \
  --disable-pinentry-emacs \
  --disable-inside-emacs \
  --disable-pinentry-gtk2 \
  --disable-pinentry-curses \
  --enable-pinentry-tty

When you run this command the output should be:

Pinentry v1.1.0 has been configured as follows:
[output omitted]
Default Pinentry .: pinentry-tty

Finally Build GnuPG

Once all the packages are done you should be ready to finish building GnuPG:

cd /usr/local/src/gnupg-2.2.9
./configure
make
make check
make install

dirmgr

When I tried to use --recv-keys or --send-keys I got this error:

gpg: failed to start the dirmngr '/usr/local/bin/dirmngr':
No such file or directory

dirmgr is used by GnuPG to perform network operations. It’s actually part of the gpg package and has already been compiled and linked when you built it. The only step missing is installation, you do need to use sudo in this case:

cd /usr/local/src/gnupg-2.2.9/dirmngr
sudo make install

Presenting at RailsConf 2017

October 28, 2017

One of the benefits of working at ActBlue is that you have the opportunity to play with lots of traffic.

In August of last year, I gave a talk at Boston Ruby Group sharing our experience building a high performance web application. The feedback I received from that audience and from colleagues allowed me to improve it and submit a proposal to RailsConf 2017.

My application was accepted in the High Volume track. I gave my talk last April, and although I was pretty nervous, I’m happy with the result. After 6 months, the talk is ranked #26 of 86 with 500 views in Confreaks. I recommend you to watch it.

Understanding Unicode Encoding in Ruby by Example

April 25, 2017
Unicode es fantástico

Every time I have to troubleshoot a problem with Unicode, it takes time to go through the documentation.

I compiled this list of methods and examples of how to use them. It has proven to save me time by quickly refreshing my memory.

Tested with Ruby 2.4.1 on macOS Sierra 10.12.4.

# the encoding is a property of String
utf8_resume = "Résumé"
=> "Résumé"
utf8_resume.encoding
=> #<Encoding:UTF-8>

# translate the same string to different encodings
latin1_resume = utf8_resume.encode("ISO-8859-1")
latin9_resume = utf8_resume.encode("ISO-8859-15")
utf8_resume.encoding
=> #<Encoding:UTF-8>
latin1_resume.encoding
=> #<Encoding:ISO-8859-1>
latin9_resume.encoding
=> #<Encoding:ISO-8859-15>

# specify the string using codepoints
lower_spanish_accents = "\u00E1\u00E9\u00ED\u00F3\u00FA\u00F1".encode("UTF-8")
=> "áéíóúñ"
upper_spanish_accents = "\u00C1\u00C9\u00CD\u00D3\u00DA\u00D1".encode("UTF-8")
=> "ÁÉÍÓÚÑ"

# Length of the encoded text

# in UTF-8
#   'z' is 1 byte
#   'ñ' is 2 bytes
z = "\u007A"
=> "z"
z.each_byte.map{|c| "%X" % c}
=> ["7A"]
n_tilde = "\u00F1"
=> "ñ"
n_tilde.each_byte.map{|c| "%X" % c}
=> ["C3", "B1"]
n_tilde.bytesize
=> 2

# but in Latin-1 'ñ' is only 1 byte
n_tilde.encode('iso-8859-1').each_byte.map{|c| "%X" % c}
=> ["F1"]

# but Unicode is universal, so in codepoints there's no difference
# between UTF-8 and Latin-1
n_tilde.each_codepoint.map {|c| "%X" % c}
=> ["F1"]
n_tilde.codepoints.size
=> 1
n_tilde.encode('iso-8859-1').each_codepoint.map{|c| "%X" % c}
=> ["F1"]
n_tilde.encode('iso-8859-1').codepoints.size
=> 1

# codepoints are base-10 integers
n_tilde.encode('iso-8859-1').codepoints
=> [241]
n_tilde.codepoints
=> [241]

# formats to specify codepoints

# single codepoint
# exactly 4 hex digits
#   \uXXXX            <==> U+XXXX
# multiple codepoints
# hex digits
# leading 0 is optional
#   \u{X XX XXX XXXX} <==> U+000X U+00XX U+0XXX U+XXXX
"\u0045\u0073\u0070\u0061\u00F1\u0061"
=> "España"
"\u{45 73 70 61 F1 61}"
=> "España"

# sometimes codepoint and byte sequence will match
"\u007f".each_codepoint.map{|c| "%X" % c}
=> ["7F"]
"\u007f".each_byte.map{|c| "%X" % c}
=> ["7F"]

# but this isn't always true
# see also example for ñ above
"\u0080".each_codepoint.map{|c| "%X" % c}
=> ["80"]
"\u0080".each_byte.map{|c| "%X" % c}
=> ["C2", "80"]

# not all byte sequences are valid encodings
"\u3042".valid_encoding?
=> true
"\u3042\x81".valid_encoding?
=> false

# scrub to the rescue
scrubbed = "\u3042\x81".scrub('')
scrubbed.valid_encoding?
=> true
scrubbed.each_codepoint.map{|c| "%X" % c}
=> ["3042"]

# building the string using the internal representation, i.e. byte by byte
espana_utf8 = [0x45, 0x73, 0x70, 0x61, 0xc3, 0xb1, 0x61]
=> [69, 115, 112, 97, 195, 177, 97]
espana_utf8.pack('c*').force_encoding('utf-8')
=> "España"

# now in Latin1 (different byte sequence)
espana_latin1 = [0x45, 0x73, 0x70, 0x61, 0xf1, 0x61]
=> [69, 115, 112, 97, 241, 97]
espana_latin1.pack('c*').force_encoding('ISO-8859-1')
=> "Espa\xF1a"
# although the ñ doesn't look correct, the enconding is correct
espana_latin1.pack('c*').force_encoding('ISO-8859-1').valid_encoding?
=> true

# currency symbols in UTF-8
currency_utf8 = "\u{20AC A3 A5}"
=> "€£¥"

# Convert a number from any base to any base
class String
  def convert_base(from, to)
     to_i(from).to_s(to)
  end
end
# example: letter "~", from base 16 to base 10
"7E".convert_base(16, 10)
=> "126"
# example: decimal 255 to hexadecimal
'255'.convert_base(10, 16)
=> "ff"

Politics and Twitter Popularity

April 19, 2016

As an engineer of ActBlue I have been following closely the Sanders campaign. Leaving aside his political views, he is a unique candidate leading a historical effort, as the first person to run for president of the USA funded exclusively by small donations.

It is a fact that no political career is viable without the backing of millions of dollars. Citizens United made things worse by allowing unlimited election spending by individuals and corporations. The result is a few affluent people exerting a great deal of power in government.

ActBlue is trying to change this by providing a technology platform that allows the collection of money from ordinary people fast and efficiently. Although ActBlue has existed for over 10 years, Bernie is the first presidential candidate to rely on it as his only source of funding.

Sanders announced in April 30 of last year he was running for president and raised one million dollars in small donations in the first day. During the following 4 months, from May to August, he raised $5 million monthly.

Most people had never heard of Bernie before and challenging someone as popular as Hillary Clinton in the primaries was a little crazy. In August 22nd Hillary had 4 million Twitter followers compared to 590 thousand for Bernie, a huge difference. That is when I decided to write a script and record the number of followers daily for the most popular candidates. It would be a fun little project to see how the numbers evolved as the primaries progressed.

Obviously Twitter popularity does not translate to votes, but some correlation makes sense. To me more important than the follower count itself is its growth rate, how quickly each account is getting new followers.

The only candidate as popular as Hillary was Trump. So I am separating the data in 2 groups, the Multi Million Group with Hillary and Trump and the Single Million with everyone else.

Trump starts with fewer followers but surpasses her after the first Democratic debate in October 13. Growth is clearly on Trump’s side and in my opinion the graph shows that a face off between the two in a general election does not look for Clinton.

It would be a mistake to underestimate the entertainer. Ronald Reagan not only was elected, he is the best president in US history according to Republicans.

In the Single Million Group I am including additional candidates because I wanted to see their behavior after they had dropped out of the race. My apologies to any Kasich supporter reading this for not tracking his numbers.

It is interesting to see what happened on the days of large increases. Carson has the largest rise after the fourth Republican debate (Nov 10). All the candidates see a jump the day of the first Democratic debate (Oct 13). There are also big increases on the second Republican debate (Sep 16).

Ted Cruz has the highest steady rate of increase among Republicans.

Bernie leads on growth with his two accounts: BernieSanders and sensanders, each quadrupled its followers.

And that is all I have, 8 month of data until today, April 19. I am not going to make any predictions. I do not believe anyone can really foresee what is going to happen, even with better information, like poll numbers.

Although half of the states have voted we do not have a clear Democratic winner yet. Bernie has won 8 of the last 9 state primaries, and today is the turn of New York. Hillary has 1,307 delegates and Bernie 1,094 (2,383 are necessary to win the nomination).

Sanders not only has been able to sustain a campaign funded by small donations, he has actually raised more money than any other candidate: 7 million contributions with an average of $27 each, that is $189 millions.

I will keep recording the counts and will post new graphs in the future. In the meantime we, the engineers at ActBlue, are enjoying the challenge of making sure we take all those donations without any disruptions.

Because of the volume, this requires a continuous effort to improve performance. Our current record is from New Hampshire primary night, when Sanders gave his victory speech. He asked people to visit berniesanders.com and donate $27. Our traffic spiked to 333,000 requests per minute, at some point we were processing 44 credit cards per second. That’s right, we were feeling the bern.

The scripts to gather the data, process it and generate the graphs can be found in github