A geek stranded on Martha’s Vineyard

Politics and Twitter Popularity

April 19, 2016

As an engineer of ActBlue I have been following closely the Sanders campaign. Leaving aside his political views, he is a unique candidate leading a historical effort, as the first person to run for president of the USA funded exclusively by small donations.

It is a fact that no political career is viable without the backing of millions of dollars. Citizens United made things worse by allowing unlimited election spending by individuals and corporations. The result is a few affluent people exerting a great deal of power in government.

ActBlue is trying to change this by providing a technology platform that allows the collection of money from ordinary people fast and efficiently. Although ActBlue has existed for over 10 years, Bernie is the first presidential candidate to rely on it as his only source of funding.

Sanders announced in April 30 of last year he was running for president and raised one million dollars in small donations in the first day. During the following 4 months, from May to August, he raised $5 million monthly.

Most people had never heard of Bernie before and challenging someone as popular as Hillary Clinton in the primaries was a little crazy. In August 22nd Hillary had 4 million Twitter followers compared to 590 thousand for Bernie, a huge difference. That is when I decided to write a script and record the number of followers daily for the most popular candidates. It would be a fun little project to see how the numbers evolved as the primaries progressed.

Obviously Twitter popularity does not translate to votes, but some correlation makes sense. To me more important than the follower count itself is its growth rate, how quickly each account is getting new followers.

The only candidate as popular as Hillary was Trump. So I am separating the data in 2 groups, the Multi Million Group with Hillary and Trump and the Single Million with everyone else.

Trump starts with fewer followers but surpasses her after the first Democratic debate in October 13. Growth is clearly on Trump's side and in my opinion the graph shows that a face off between the two in a general election does not look for Clinton.

It would be a mistake to underestimate the entertainer. Ronald Reagan not only was elected, he is the best president in US history according to Republicans.

In the Single Million Group I am including additional candidates because I wanted to see their behavior after they had dropped out of the race. My apologies to any Kasich supporter reading this for not tracking his numbers.

It is interesting to see what happened on the days of large increases. Carson has the largest rise after the fourth Republican debate (Nov 10). All the candidates see a jump the day of the first Democratic debate (Oct 13). There are also big increases on the second Republican debate (Sep 16).

Ted Cruz has the highest steady rate of increase among Republicans.

Bernie leads on growth with his two accounts: BernieSanders and sensanders, each quadrupled its followers.

And that is all I have, 8 month of data until today, April 19. I am not going to make any predictions. I do not believe anyone can really foresee what is going to happen, even with better information, like poll numbers.

Although half of the states have voted we do not have a clear Democratic winner yet. Bernie has won 8 of the last 9 state primaries, and today is the turn of New York. Hillary has 1,307 delegates and Bernie 1,094 (2,383 are necessary to win the nomination).

Sanders not only has been able to sustain a campaign funded by small donations, he has actually raised more money than any other candidate: 7 million contributions with an average of $27 each, that is $189 millions.

I will keep recording the counts and will post new graphs in the future. In the meantime we, the engineers at ActBlue, are enjoying the challenge of making sure we take all those donations without any disruptions.

Because of the volume, this requires a continuous effort to improve performance. Our current record is from New Hampshire primary night, when Sanders gave his victory speech. He asked people to visit berniesanders.com and donate $27. Our traffic spiked to 333,000 requests per minute, at some point we were processing 44 credit cards per second. That's right, we were feeling the bern.

The scripts to gather the data, process it and generate the graphs can be found in github

How to Upgrade to Strong Parameters in Rails

December 10, 2014

This article was originally published in the blog of ActBlue Technical Services.

How to Upgrade to Strong Parameters in Rails

In the previous blog post, using a series of tests, we described how strong parameters work. In this post we detail the steps we followed to upgrade our main application in ActBlue.

Starting Point

To help with the process Rails provides the protected_attributes gem. This allows you to run your app on 4.1 without having to make any changes related to mass-assignment protection. The gem brings backwards compatibility by implementing attr_accessible, attr_protected and other methods.

Add these lines to your Gemfile, run bundle install and make all your tests pass (you have tests, right?)

gem 'rails', '~> 4.1'
gem 'protected_attributes'

There are other incompatibilities you will have to resolve, but this post is about strong parameters and we are assuming your tests run green at this point.

Mixing Both

It is unlikely you can make all the changes in a single release and therefore you will want to have both protected attributes and strong parameters working at the same time. So the next step is to add this line to Gemfile and run bundle install:

gem 'strong_parameters'

In every model you want to upgrade add this line to include the ForbiddenAttributesProtection module. This is the way to indicate Rails which models are using the new mechanism, for example:

class Book < ActiveRecord::Base
  include ActiveModel::ForbiddenAttributesProtection
end

In the model also remove all calls to attr_accessible and attr_protected, for example:

attr_accessible :isbn, :title
attr_protected :price

In the corresponding controller it is useful to create a private method that whitelists its parameters, for instance:

private
def book_params
  params.require(:book).permit(:isbn, :title)
end

For simple models and controllers it is going to be similar to this, but for more complex cases you can check our previous blog post or the documentation.

Dropping Protected Attributes

When you think you have upgraded all your models and controllers and feel you are ready to pull the plug on the old protected attributes, these are the steps we recommend:

Remove these 2 lines from Gemfile and run bundle install.

gem 'protected_attributes'
gem 'strong_parameters'

In Rails 4 the default is strong parameters, so there is no need to include the gem.

Remove the configuration for protected_attributes, this means removing from config/application.rb:

config.active_record.whitelist_attributes = false

Remove from config/environments/test.rb:

config.active_record.mass_assignment_sanitizer = :logger

Remove any include ActiveModel::ForbiddenAttributesProtection from models, added in the previous step.

In the default configuration of strong parameters, when you whitelist only a subset of the params passed to the controller, instead of raising an exception, Rails is going to generate a notification. You can see a test illustrating this.

It is better instead to always raise an exception. This is done by adding this line to config/environments/test.rb:

config.action_controller.action_on_unpermitted_parameters = :raise

Now run all your tests. If you have a large application and you are getting errors you do not understand or the code is not behaving the way you expect, check our previous blog post Understanding Strong Parameters, this is the part where this information is most useful.

Final Step

When you are done with the tests you can remove the line from config/environments/test.rb:

config.action_controller.action_on_unpermitted_parameters = :raise

The default value in Rails is to generate a notification in test and development environments, and ignore otherwise.

You are done! We hope the 2 articles helped you to make the transition smoother.

Understanding Strong Parameters in Rails 4

November 14, 2014

This article was originally published in the blog of ActBlue Technical Services.

Understanding Strong Parameters in Rails 4

Earlier this year, in ActBlue, we completed the upgrade of our main application from Rails 3.2 to 4.1. The last step in the process was incorporating strong parameters.

Several of our models are simple and we did not have a problem following the documentation, RailsCast and other articles. But a good number of models (and its controllers) are more complex or were using our own protection mechanism, and we were dragging our feet on them.

The main issue was that we did not really understand how strong parameters worked. Correction: I did not understand, which is the big realization that took place when I had to explain what I was doing to the rest of the team. The part that is missing from all the articles I read is what happens when you do not use strong parameters in the correct way. For this reason, the best solution was a series of tests, which is what we present here.

In order to minimize the dependencies, the tests were written in MiniTest::Spec, and the only additional gem required is mocha. Ruby version is 2.1.4 and Rails 4.1.6.

The full list of tests can be found in this github repo, following we go through the most important ones:

If you change the version of Rails in your Gemfile from 3.2 to 4.1, without making any other changes, parameters in the controllers will be passed untouched to methods new and update_attributes on your models, raising this error:

describe 'completely unhandled params (i.e. no call to permit)' do
  before do
    @hash = { 'title' => 'Test', 'body' => 'test body' }
    @params = ActionController::Parameters.new @hash
  end
  it 'should raise error' do
    -> { Article.new @params }.must_raise ActiveModel::ForbiddenAttributesError
  end
end

The next step is to start whitelisting some parameters by calling the permit method. For sure, you will miss some, which will trigger this notification:

describe 'permit some elements from params and miss others' do
  before do
    @params = ActionController::Parameters.new @hash
  end
  it 'should log non-permitted attributes' do
    ActiveSupport::Notifications.
      expects(:instrument).
      with('unpermitted_parameters.action_controller', keys: ['body']).
      once
    @params.permit(:title).must_equal 'title' => 'Test'
  end
end

In Rails 3.2 when you create or update objects on a model and include attributes that do not exist they willl just be ignored. Version 4.1 raises an exception, which is a better way of handling it, no one would ever rely on the fact they are ignored, right?

describe 'call permit on attributes that do not exist in the model' do
  before do
    params = ActionController::Parameters.new @hash.
      merge('non_attr' => 'test')
    @permitted_params = params.permit(:title, :non_attr)
  end
  it 'ActiveRecord should raise exception on non-attribute' do
    ex = -> { Article.new @permitted_params }.
      must_raise ActiveRecord::UnknownAttributeError
    ex.message.must_equal 'unknown attribute: non_attr'
  end
end

The require method is useful when you want to make sure that certain parameter is present, if not it will raise an exception:

describe 'require something that is not present in params' do
  before do
    hash = {
      'article_attributes' => {'title' => 'Test', 'body' =>'test body'},
      'other_attributes' => {'amount' => '12', 'color' => 'blue' }
    }
    @params = ActionController::Parameters.new hash
  end
  it 'should raise exception' do
    ex = -> { @params.require(:category_attributes) }.
      must_raise ActionController::ParameterMissing
    ex.message.must_equal 'param is missing or the value is empty: ' +
                          'category_attributes'
  end
end

Note that if there are other keys in the params hash they will be removed:

it 'should filter out everything but the required key' do
  @params.require(:article_attributes).must_equal 'title' => 'Test',
                                                  'body' =>'test body'
end

Additionally, requiring without calling permit will raise an exception:

it 'should raise error if we try to use it as is (without permit)' do
  -> { Article.new @params.require(:article_attributes) }.
    must_raise ActiveModel::ForbiddenAttributesError
end

Finally the “standard” way to use strong parameters, this is the form that you will see presented in other blog posts.

describe 'proper use' do
  before do
    @attributes = @params.require(:article_attributes).
      permit(:title, :body)
  end
  it 'should be fine if we use require and permit combined' do
    @article = Article.new @attributes
    @article.title.must_equal 'Test'
    @article.body.must_equal 'test body'
  end
end

Nested attributes on a model that has a one-to-one relation. In the example a comment belongs_to an article:

it 'shows how to permit nested params' do
  attributes = @params.
                  require(:comment).
                  permit(:author, article_attributes: [:id, :title,
                                                       :body])
  attributes.must_equal "author" => "John Smith",
                        "article_attributes" => {
                          "title" => "Test",
                          "body" => "test body"}
  comment = Comment.new attributes
  comment.author.must_equal 'John Smith'
  comment.content.must_be_nil
  comment.article.title.must_equal 'Test'
  comment.article.body.must_equal 'test body'
end

Nested attributes on a model with a one-to-many relation. The example shows an article that has_many comments. Note that the controller will receive an extra level of keys, the integers 0, 1, etc. Rails internally ignores these numbers so we don't have to include them in the call to permit:

describe 'nested attributes on articles (which has_many comments)' do
  before do
    @hash = {
      "article" => {
        "title" => "Test",
        "body" => "test body",
        "comments_attributes" => {
          '0'=>{"author" => "John", "content" => "great"},
          '1'=>{"author" => "Mary", "content" => "awful"}
        }
      }
    }
    @params = ActionController::Parameters.new @hash
  end
  it 'shows how to permit nested parameters' do
    attributes = @params.
      require(:article).
      permit(:title, comments_attributes: [:author,
                                :content])[:comments_attributes].
        must_equal @hash['article']['comments_attributes']
  end
end

The repo has additional cases and also shows how the code looks when you are doing updates.

In the next blog post will show the specific steps we took during the upgrade of our application.

Introduction to Contributing to Open Source

September 02, 2013

This is the mini workshop I will be giving today at Project Night, the event organized by Boston Ruby Group and hosted by thoughtbot every first Tuesday of the month.

Many people starting in software development feel they would like to contribute to open source but they do not know how to do it. In this class we will review the basic workflow to make a change to an open source project. We will learn how to navigate github, how to open a ticket and how to send a pull request, the transaction by which we implement the update. The interaction that takes place between contributors and project owners will also be covered.

The open source project we will be working on is the RailsBridge Boston website, of which I am an owner. This way students will be able to have a real world experience with confidence, without fear of making mistakes.

Requirements

You must have a GH account with SSH keys. If you do not have it, follow these steps:

Workflow

This is a diagram of the workflow we will be following during the class.

github flow

Concepts

Repository
a project in git/github, i.e. a bunch of files
Issue
ticket, conversation about bug, feature, idea
Commit
snapshot of a repository
Pull request
set of commits

Steps

  • Login to your github account.

  • Search for railsbridge boston, click on users and select that entry.

  • Select the repo railsbridge-boston.

  • Read the README.

  • Visit the production version of this website: http://railsbridgeboston.org/.

  • Click through this sequence in the top menu: Curriculum > Friday > Installfest > Install Ruby on Windows. This is the page we will be modifying.

  • Go back to github, find the list of commits and look for one from 7/2/13 called “Fixed typo on homepage”.

  • Click on this commit and check its content. It was done by a student of this same mini workshop two months ago. Note that although a commit is a snapshot of all the files we only have to handle the differences.

  • Check other commits on 7/2 and 7/6, note the granularity of the updates.

  • Make a fork of this repo. Check the diagram above to understand what happens when you perform this operation.

  • Make sure you are on the github page of your own repo and not railsbridge-boston.

  • Clone this repo, in the command line:

git clone git@github.com:myuser/railsbridge-boston.git
cd railsbridge-boston
  • Because this might take a few minutes, we will start doing something else. Leave this tab open in your browser, this is the project we are working on.

  • Go to the github page for the railsbridge-boston user and select the repo railsbridge-virtual-machine. This page contains the instructions we want to use to install a Windows machine, i.e. this is what we want to use instead of RailsInstaller in the next RBB workshop.

  • Create an issue saying that the instructions should be modified.

  • The project owner will show on the projector how the issue you just filed looks from his/her side and he/she will submit a response.

  • Review the thread of the conversation in your computer.

  • Go back to the git clone you left running.

  • Find the file in your laptop that contains the page you need to modify. This can be done with the grep command:

grep -r 'Installing Ruby and Rails on Windows is easy thanks to' *
  • Open your text editor and make the changes.

  • Before sending your update to github, review the changes you just made. Use these 2 commands:

git status
git diff

Push the changes to github. Check the diagram to understand what is happening.

git add .
git commit -m 'Use Vagrant instead of RailsInstaller'
git push origin master
  • Go to the page with your repo on github and make a pull request.

  • Watch the side of the project owner, how she makes a comment on the PR and also a comment under a specific line of code.

  • Experiment with the interaction between contributors and owners.

  • Watch how the project owner merges the PR and deploys into production.

  • Check that the update is visible in the production website.