Optimist's Guide to Pessimistic Library Versioning

Upgrading software is much harder than it could be. Modern versioning schemes and package managers have the ability to help us upgrade much more than they do today.

Take for example my post on Upgrading to Rails 5 Beta — The Hard Way. Most of the time was spent trying to find all the different libraries my app was using that weren’t compatible yet with Rails 5 and upgrade them.

What if somehow bundle update rails could have known they didn’t work and instead find a version that did work? It turns out that it can, but it requires some pain on the maintainer’s part. In this post, we’ll look at different strategies for declaring dependencies in libraries, why one is dominant right now, and how we might be able to make major version bumps easier into the future.

Keep Reading on Codeship

Get Your Conference Proposal Accepted

Want to speak at a conference? First you'll have to get a proposal accepted. To date, I've helped pick talks for Keep Ruby Weird since it started, and this year I reviewed for RailsConf 2016. I've also written a number of proposals, some that have been accepted, some that haven't.

There's no sure fire formula to guarantee your talks get accepted. In this post I'll share some of the things I liked and disliked while reading proposals as well as some of the things that I've found help when writing my own proposals.

I'll be talking about things I didn't like in proposals, however all examples are made up. Any likeness to existing proposals is coincidental. When possible examples are given from my own proposals.

RailsConf 2016

I only recently started taking notes while reviewing proposals for RailsConf. There were over 500 to read through, and at some point, many start blending together into a generic proposal blob. I'll mostly be focusing on this CFP experience. The CFP is split into 4 parts - title, abstract, details, and pitch. The App used for the CFP is open source. From the README:

The heart of the proposal is the title and abstract. These two will show up in your program and schedule. They are required fields and have limits on how long they can be. The details field is for reviewers/organizers only and is there to allow the speaker to show 'what's behind the curtain' like an outline or other explanations that don't belong in the abstract. The pitch is there to allow the speaker to sell you on why this talk should be in your event.

Let's start off simple. Try not to make overt typos in anything you submit. They stick out like a sore thumb in otherwise potentially well written submissions. It seems trivial, but when your talk comes up as the 70th submission that a reviewer is looking at that day it might break their brain and hurt your chances. If you're not a native English speaker or you're awful with grammar like me, ask a friend for a review. I'll also contract with editing services: Draft's "Ask a Pro" feature is quick and easy.

Pitch

The first round of reviews is blind, however in the "Pitch" section some people are able to give lots more specific details without identifying themselves than others. I found myself leaning towards reading the "Pitch" section first as it was one area where some talk proposals really stood out. To me this section was all about the submitter's credibility. Why should I listen to them? Are they an amazing speaker and will give a great performance? Are they an expert in a field that they can speak authoritatively on? Do they have some unique experience that makes their content special or different? You can't identify yourself here, but you can identify traits or facts about yourself. Did you win "best comedic presentation" at your local toastmasters? Were you in every theatre play and musical production in school? Did you write an open source tool or library that deals heavily with the topics involved? Have you spoken before? It doesn't have to be at a conference, tell me about your love for speaking at school or user groups, something that indicates your general ability to give a well thought out and prepared talk. Where did you speak? How was it received? What audiences did you resonate with? Did you write a book? Where do you work? Do you hate with a passion due to ? How did you find the topic you're submitting about, and what makes you the best person to speak on it?

Another way to think of a pitch section is if two people submitted otherwise identical proposals, how would you sell yourself as being the best candidate to speak?

A Cliche is Like...

Don't use cliches, everyone use's cliches and it makes your proposal blend in. If you feel the pull of using a phrase everyone has heard "a million times", ask yourself if it adds to what you're trying to convey. For getting out of this habit I recommend Word Hero. Along the line of cliche's avoid absolutes in your writing unless you know they're true

  • "Everyone's heard of": I haven't heard about this before, am I nobody?
  • "Obviously": If it's so obvious, why did you have to write it? Drop the word.
  • "We've all heard": Then it probably isn't worth speaking about.

These types of "absolute" phrases only work when you really know your audience, even then they sound weak. Not to be "beating a dead horse" and absolutely NEVER use "absolute" phrases.

Details

The details section is where you can show how much you've thought about your topic. Did you just wake up one morning and think "hmm, maybe I could talk about Texas BBQ sauce history" or is this something that's been brewing in the back of your brain for the last six months? The details section is where I'm interested in specifics. Facts, figures, and concepts. When I read it, I want to form a mental picture of someone walking around the stage going from topic to topic building towards a central argument, or whatever the point of the talk is.

BTW, your talk should have a point. When someone walks away from your talk they might only take away 2-3 things with them. Those things should be in your proposal, the details section seems like a reasonable place to put them. As much as I like to see specific content in the details section, I don't want 60 bullet points of in depth takeaways. You only have <30-60> minutes depending on the conference and talk slot. 60 bullet points leaves you with 30 seconds per each, give your topic room to breathe. Don't give a comprehensive overview of EVERYTHING in a field, you won't have time. Go deep in one part.

If you know what you want to talk about, but you're not sure how you want to structure it, check out Jeff Casimir's great 'Prototypical Conference Talks' it has some do-s and some don'ts.

Medium over Message

One big deciding factor for me on whether a talk should be accepted or not, is asking "is this the right medium for this content". Before you submit a proposal on a topic, consider writing a blog post about it. If you were able to say everything you wanted to in 1 post, then it's probably not a good candidate for a talk. Written words work well for reference and getting started, materials, and for comprehensive rundowns. Talks on the other hand favor emotional communication. What is the story behind this refactoring? Are you taking a boring topic and making it interesting or memorable somehow in a way that only a live performance could do justice to? Imagine your talk is finished and has been recorded by Confreaks. Under what circumstances would someone say "hey you should watch a video about " and link to your video? If you see a high possibility of that happening, then your topic is well suited for the stage.

The detail section is where you can also get into sharing your personality as it relates to the topic. What are the themes? Are there jokes you plan to tell? How will the things that make you unique co-mingle with your talk topic? Statements like "This is the talk I wish I could have attended a year or two ago" are great if they lead to helping the reviewer uniquely understand your position on a topic. How does your talk make the world a better place for the people who come to see it?

Something that I liked when consuming the details section is a bullet point list that would show the general flow of information and ideas. Something I didn't like is when proposals went so far as to give minute markers telling how long they would spend talking about each thing. Estimates are okay but I don't want it to look like a stenographer's transcript, it's mind numbing.

Did I mention I feel like talks are a good time to share information through stories? If you think the pitch section is about selling yourself and your credibility, then the details section is about selling your topic presentation flow, takeaways, and the stories that bind them all together.

Abstract

Imagine you bump into me in an elevator, you're about to go give your talk. I ask if I should go, you have 400 words to sell me on attending. What would you say? For many conferences the title and abstract are the only things they ask for. I'm seeing more and more organizers seeking more info like RailsConf does. Either way you'll have to write one, and the better it is, the better your chances are of getting accepted.

An abstract to me is a pure marketing pitch. You can google on how to "write good copy" and see a ton of "do this" and "don't do that" lists. It may sound cheesy but many of the techniques work well at grabbing attention and keeping it. Start off with a bang, if your first sentence isn't good they won't read the rest. Here's a few of my opening sentences of talks that I've gotten accepted:

  • What do you do when a maintainer leaves a project with over 44 million downloads?
  • Underneath the beautiful veneer of our Ruby libraries lies a twisted tangle of writhing guts.
  • Heroku has deployed millions of web apps.
  • Good tests are isolated, they’re repeatable, they’re deterministic.
  • Level up your programming game, and change the world.
  • Run your app faster, with less RAM and a quicker boot time today. How? With science!
  • If you've ever misspelled a word while searching on Google, you've benefited from decades of distance algorithm development.

I'm not going to claim they're all master works, but they're at least kinda interesting. Another key point in abstracts is brevity. Use short sentences. Short sentences are concise. Short sentences are rich. Short sentences are powerful. Short. Sentences.

Remove unnecessary words, obviously. Remove soft or hedging statements "sometimes", "maybe", etc. It's okay to be a little hyperbolic in an abstract. Instead of "your tests might be slower than you may like" it's okay to shorten to "your tests are slow". Attendees will self select so that statement will likely be true. If my tests are fast, i don't need your talk. Being concise in an abstract shows me that you know the core essence of what you want to talk about and you know how to extract and distill it.

Once you've grabbed attention, you'll want to make sure to give your reader enough information to determine whether or not they should see the talk. Is it an advanced topic? Very technical? Any prerequisites? I like being specific on these points. Here's some of my "in this talk" statements from proposals that have gotten accepted over the years:

  • In this talk we'll break down some popular distance measurements and see how they can be implemented in Ruby. We'll look at real world applications with some live demos.
  • You will learn how to stay sane in the face of insane testing conditions, and how to use these tests to deconstruct a monolith app.
  • We will cover what it takes to go from no-grammer to a senior level programmer and everything in between. Don’t come to this talk if you don’t want to be motivated, you don’t want to succeed, and certainly if you don’t want a good swift kick in your Ruby programming pants.
  • We'll look at real world use cases where small changes resulted in huge real world performance gains. You'll walk away with concrete and actionable advice to improve the speed of your app, and with the tools to equip your app for a lifetime of speed.
  • In this talk you'll learn how to use simple and sharp Ruby tools to slice into large libraries with surgical precision. Turn your impossible bugs into pull requests, and level up your programming skills by Dissecting Ruby with Ruby.
  • We've taken a look at the most common patterns and boiled down the best of our advice in to 12 simple factors that can help you build your next app to be stable, successful, and scalable. After this talk you'll walk away with in depth knowledge of web framework design patterns and practical examples of how to improve your application code.

Tell me what I can expect to learn, what I can expect to take away. It's not a requirement, but it helps to cut some of that marketing hyperbole with some straight talk.

A well written abstract is almost like a piece of technological poetry. It is more than the sum of it's words. Speaking of words one thing I found helps in my abstract creative process, is to make word clouds. Given a topic like "security" or "performance" I write down as many other words come to mind. I then stack rank them based on which are the most interesting. I write about 10-20 different intro sentences. About 3-4 different copies of the "takeaway" sentences, and about 5-10 outro sentences. I then mix them up until they feel right.

When working on an abstract get feedback, from anyone. How does it make them feel? What do they think the talk is about? What types of things would they consider about the abstract before deciding to come or not? If they respond "good", that's not helpful press them for details. It helps if that someone has submitted to conferences, but it isn't required. If you find yourself working with experienced submitters, don't let them re-write or edit out your voice. Don't force a sentence in that won't fit just because it sounds great on its own. Write lots, throw away lots.

Title

The best wisdom for writing titles is to save it for last. I hate this advice because I always forget about it until I've spent 4 hours up front trying to write a good title. Instead my advice is set a timer for 3 minutes, write as many titles as you can. Pick your favorite, then write the rest of your submission. When you're done go back and throw away the title and re-write it. Chances are your original title won't match your final talk concept. Try to spend a bit more time writing down title ideas, maybe 10-15 minutes. Try testing your favorite 2-4 on a few people and then go with your gut as to the best one.

Try to use descriptive words or action statements in the title. Really avoid cliches here, I'm really not interested in "how [you] stopped worrying and learned to love the <blank>".

Don't alienate your audience through jargon. Your title should say something about your talk. If it's an "in" joke or an acronym don't use it. I get it that you know what HATEOS is, but some of your attendees or conf selection committee might not. (No one used this acronym in this year's CFP to my knowledge I'm not picking on anyone's proposal).

As a rule don't put anything in your title or abstract that your attendee would not understand unless they already attended. You can lean on your knowledge of the conf, it's likely that attendees of RailsConf will have heard of REST, so you can probably use it, but unless you're talking explicitly about that topic maybe it's not super helpful. When in doubt lean on being more descriptive, and less creative. You can also use two titles with a colon if it helps. I did this for "Millions of Apps: What we've Learned". The title is probably the least important thing to me as a reviewer, but likely the most important thing to an attendee in terms of picking your talk. Spending time working on your "pitch" and your "details" actually can help improve the quality of talk you give, spending hours on your title won't. Budget your time accordingly, don't dwell on the perfect name™.

Submission notes

If you want to increase your chance of getting a talk accepted I have two pieces of advice.

1) Submit more than one talk to each conference 2) Submit to 3x the number of conferences you want to speak at

Speaking is a numbers game. The great players in baseball only hit the ball 1/3 of the time at best. I find I only get accepted to every 1 out of 3 conferences I apply to. There's a zillion reasons why your talk wasn't accepted that has nothing to do with you.

  • Too similar to another already accepted
  • Too different to the other talks accepted
  • Technical, and they needed more non-technical content
  • Too advanced
  • Too geared towards beginners
  • All over the place
  • Too specific
  • Not the right fit for attendees
  • Your talk isn't unique (gave at other confs)
  • Your talk is too immature (haven't given at any other confs)

The list goes on. Trying to predict what a conference needs and writing to that is impossible. Most reviewers don't even know until they have read all the proposals. I try to at bare minimum always write one technical and one non-technical leaning talk. I like to have 3 or so proposals in my back pocket that I would be willing or interested in presenting on. Writing 3 proposals takes 3 times as long, but they can be reused for other confs. When I am in a writing mood, it is easier to write 3 talks than to try to find 3 separate times. If you find writing a specific proposal idea is too hard or painful, it's a good sign that isn't a great topic to talk on, imagine having to expand that proposal to a full 30 minutes.

Lots of good talks get rejected to every conference. Rejection never gets any easier. My one piece of advice is to focus on the future of submitting to other confs. The reviewers likely won't have time to give you specific feedback on why a talk wasn't accepted. Even if they do you might not like what you hear, it probably won't be actionable to future talks. I've never been given feedback from conference organizers. Instead I focus on bright spots.

Look at the talks that were accepted, what did they have in common with your proposal? What did they have that was different? Maybe you can ask some of the people who got talks accepted to share their details & pitch (things that are not put on the schedule). They don't have to, of course, but if they do it is more data for you. Don't be snarky about it though, show genuine appreciation and interest. "I saw your abstract on the program and thought it sounded really interesting, I definitely want to go. I'm trying to get better at submitting to conferences would you be willing to share your 'details' and 'pitch' in a public blog post or a private gist?".

Again, it may have been blind luck that their talk was chosen, or maybe they did something implicitly different than you, or maybe it was bad luck that your talk wasn't chosen. The only thing you can do is try to get more data points, just don't try to read too much into each of those. Here's the three proposals I submitted this year (reviews are blind and I'm programmatically prohibited from voting on your my talks, reviewers are encouraged to submit talks). The "Saving Sprockets" was chosen, why? Who knows. Don't take these as good or bad examples, they just are. What did you like about them? What did you dislike? What have I recommended to do here that I didn't do well in my proposals?

I really want to give the "Everyday Heroes" sometime, it's more of a non-code talk that I think has really broad appeal with some useful ways to think about the world. However I seem to be better at writing proposals for more technical content. BTW. don't comment on the gist, GitHub doesn't send any notifications you can send comments via @schneems on twitter.

Your First Conf Proposal

So you've read all of this, you've decided to take the plunge. What should you do first? I think you should go speak at a user group. I help to run Austin on Rails and we practically have to hunt down people to give talks. If you approach a local user group organizer with a talk idea they'll most likely give you a slot, probably sooner than you were expecting.

If things go well, consider submitting the same talk to a conference. If they go poorly reflect on why, look for local mentors that speaks well, ask them to coffee to give you feedback on how to get better. There's lots of books, and videos on improving technical speaking. Watch good talks. Watch technical conference talks, and also watch talks in other fields like TED that are captivating. What makes them good? You don't have to present exactly like them, but did someone use a technique you really liked? Steal it, and incorporate it. Writers, write. Speakers speak. I've talked at user groups that have been standing room only 100+ people. I've spoken at major industry conferences with less than a dozen people in the room. Just because it is "local" doesn't mean it doesn't count as speaking.

In general you won't get much actionable feedback when presenting. I've had someone tell me my talk was "too advanced" and "too beginner" from different people at the same talk. If you have an in depth opinion on someone's talk, right after they gave it is not a good time. ("Hey I liked your talk" is always appreciated even if you have criticisms that you want to share later). Let the adrenaline settle, wait for a day maybe. Ask before giving advice "can I share some feedback?". Ask if there is any specific type of feedback they are looking for. Maybe they are working on the enunciation, or stage presence, maybe they feel the technical content could be tweaked.

Don't blindly give feedback, speaking is a performance. Even the best of intentioned feedback from closely trusted friends can damage egos. Try to focus on the positive, or if there was negative focus on remediation items instead of why things were bad. Speakers, the best time to get feedback is before a conference. Sit a co-worker or loved one down and present to them for an hour, get immediate feedback. Give them a pen and paper to write notes. If you're not practicing for the timing of your talk let them interrupt you with questions. You should consider baking answers of those questions into your talk. You can keep notes on a paper too. Speaking is a skill, if you practice it with appropriate feedback loops you will get better.

As an organizer of Keep Ruby Weird, my number one priority is that my audience will walk away and say "Wow, those speakers were amazing, I'm very glad I came". When we are going through the 2nd or 3rd round we don't do blind review, I have few enough speakers that I can google for videos of them presenting. A few sound bites can go a long way. It's certainly not required, but I take it as an irrefutable sign that you've at least put work into speaking as a craft. If you don't have Confreaks at your local meetup, try turning any presentation into a screencast and putting it online. Our conference doesn't penalize "non-unique" content i.e. you've given it before. We want it to be well delivered and executed, it's okay if other people have seen the talk before, it won't get moldy. It's okay if english is your second language, it's okay if the other speakers in the lineup don't look or act like you, it's okay if you don't have a famous twitter handle. A good story, well presented can be unstoppable.

Don't feel like speaking is some rite of passage every developer must go through to advance. I loved theatre in high school. I was awarded "biggest ham" in my first musical of which I had maybe 2 or 3 lines. I get a rush when I speak. I also love teaching, I was an adjunct professor at UT for a short while. I use the pressure of giving talks as a kind of peer review platform for things I'm trying to figure out. Being forced to present on a topic means I can't fake or pretend to know the subject matter. Speaking for me is as much of a personal journey as a public outreach.

I once thought that speaking was how I got to the "next level". I don't think that's true anymore. I know lots of programmers I really respect that don't speak or speak infrequently. There are other ways to express yourself, podcasts, blog posts, technical papers, writing books, pull request, GitHub issues, etc. Speaking to check off some professional bucket-list item will probably not end well. If you get up on the stage because you have something to say and you have something to get out of it, you'll be great.

Now comes the disclaimer, all these opinions and advice are mine only. I'm very biased towards my own successes and failures, try to get a diversity of opinions. Remember that they are opinions, not facts. YMMV. You can see the organizers and review team on the RailsConf about page. Maybe find them at RailsConf, thank them for reading proposals till their eyes fell out and ask them for their thoughts.

I forgot to mention, make the last sentence in your abstract somewhat memorable, preferably short, even better if it's actionable. Now go forth and write better proposals!


Richard is a junior woodworker who dabbles in programming for Heroku. He has opinions, and dad jokes. Follow @schneems on Twitter for more.

Easier Gem Releases with Bundler Release Tasks

If you maintain a gem and aren't using Bundler's release rake tasks you're missing out. If you have a well maintained gem, then the best practice is to tag a release every time you push a new gem version to RubyGems.org. This helps users to see differences between versions. For example you can compare releases on the Heroku Ruby Buildpack https://github.com/heroku/heroku-buildpack-ruby/compare/v142...v143. Bundler comes with a Rake task that simplifies tagging a release and pushing a version to RubyGems.

To use the rake tasks, you'll need a Rakefile in your project. In that Rakefile you'll need to add:

require 'bundler/gem_tasks'

You can see this line in derailed_benchmarks. Now you can see available tasks by running:

$ bundle exec rake -T
rake build            # Build derailed-0.1.0.gem into the pkg directory
rake clean            # Remove any temporary products
rake clobber          # Remove any generated files
rake install          # Build and install derailed-0.1.0.gem into system gems
rake install:local    # Build and install derailed-0.1.0.gem into system gems without network access
rake release[remote]  # Create tag v0.1.0 and build and push derailed-0.1.0.gem to Rubygems

To cut a new release, you rev the version of your gem, commit to git and then run:

$ bundle exec rake release

That will create a tag, push to GitHub and push your latest version to Rubygems. It's that easy. Using the task also ensures that the latest code you have locally is on GitHub, I've been guilty before of fixing a problem and forgetting to push to master after I cut a RubyGems release. If you're not tagging your gem release versions on GitHub, you should start. If you already are tagging manually, you can save yourself a few commands with this simple trick. Go now, I rake release you.

Speeding up Sprockets

The asset pipeline is the slowest part of deploying a Rails app. How slow? On average, it's over 20x slower than installing dependencies via $ bundle install. Why so slow? In this article, we're going to take a look at some of the reasons the asset pipeline is slow and how we were able to get a 12x performance improvement on some apps with Sprockets version 3.3+.

Originally Posted: Speeding up Sprockets on Heroku's Engineering Blog

The Rails asset pipeline uses the sprockets library to take your raw assets such as javascript or Sass files and pre-build minified, compressed assets that are ready to be served by a production web service. The process is inherently slow. For example, compiling Sass file to CSS requires reading the file in, which involves expensive hard disk reads. Then sprockets processes it, generating a unique "fingerprint" (or digest) for the file before it compresses the file by removing whitespace, or in the case of javascript, running a minifier. All of which is fairly CPU-intensive. Assets can import other assets, so to compile one asset, for example, an app/assets/javascripts/application.js multiple files may have to be read and stored in memory. In short, sprockets consumes all three of your most valuable resources: memory, disk IO, and CPU.

Since asset compilation is expensive, the best way to get faster is not to compile. Or at least, not to compile the same assets twice. To do this effectively, we have to store metadata that sprockets needs to build an asset so we can determine which assets have changed and need to be re-compiled. Sprockets provides a cache system on disk at tmp/cache/assets. If the path and mtime haven't changed for an asset then we can load the entire asset from disk. To accomplish this task, sprockets uses the cache to store a compiled file's digest.

This code looks something like:

# https://github.com/rails/sprockets/blob/543a5a27190c26de8f3a1b03e18aed8da0367c63/lib/sprockets/base.rb#L46-L57

def file_digest(path)
  if stat = File.stat(path)
    cache.fetch("file_digest:#{path}:#{stat.mtime.to_i}") do
      Digest::SHA256.file(path.to_s).digest
    end
  end
end

Now that we have a file's digest, we can use this information to load the asset. Can you spot the problem with the code above?

If you can't, I don't blame you—the variables are misleading. path should have been renamed absolute_path as that's what's passed into this method. So if you precompile your project from different directories, you'll end up with different cache keys. Depending on the root directory where it was compiled, the same file could generate a cache key of: "file_digest:/Users/schneems/my_project/app/assets/javascripts/application.js:123456" or: "file_digest:/+Other/path/+my_project/app/assets/javascripts/application.js:123456".

There are quite a few Ruby systems deployed using Capistrano, where it's common to upload different versions to new directories and setup symlinks so that if you need to rollback a bad deploy you only have to update symlinks. When you try to re-use a cache directory using this deploy strategy, the cache keys end up being different every time. So even when you don't need to re-compile your assets, sprockets will go through the whole process only stopping at the very last step when it sees the file already exists:

# https://github.com/rails/sprockets/blob/543a5a27190c26de8f3a1b03e18aed8da0367c63/lib/sprockets/manifest.rb#L182-L187

if File.exist?(target)
  logger.debug "Skipping #{target}, already exists"
else
  logger.info "Writing #{target}"
  asset.write_to target
end

Sprockets 3.x+ is not using anything in the cache, and as has been reported in issue #59, unless you're in debug mode, you wouldn't know there's a problem, because nothing is logged to standard out.

It turns out it's not just an issue for people deploying via Capistrano. Every time you run a $ git push heroku master your build happens on a different temp path that is passed into the buildpack. So even though Heroku stores the cache between deploys, the keys aren't reused.

The (almost) fix

The first fix was very straightforward. A new helper class called UnloadedAsset takes care of generating cache keys and converting absolute paths to relative ones:

UnloadedAsset.new(path, self).file_digest_key(stat.mtime.to_i)

In our previous example we would get a cache key of "file_digest:/app/assets/javascripts/application.js:123456" regardless of which directory you're in. So we're done, right?

As it turns out, cache keys were only part of the problem. To understand why we must look at how sprockets is using our 'filedigestkey'.

Pulling an asset from cache

Having an asset's digest isn't enough. We need to make sure none of its dependencies have changed. For example, to use the jQuery library inside another javascript file, we'd use the //= require directive like:

//= require jquery
//= require ./foo.js

var magicNumber = 42;

If either jquery or foo.js change, then we must recompute our asset. This is a somewhat trivial example, but each required asset could require another asset. So if we wanted to find all dependencies, we would have to read our primary asset into memory to see what files it's requiring and then read in all of those other files; exactly what we're trying to avoid. So sprockets stores dependency information in the cache.

Using this cache key:

"asset-uri-cache-dependencies:#{compressed_path}:#{ @env.file_digest(filename) }"

Sprockets will return a set of "dependencies."

#<Set: {"file-digest///Users/schneems/ruby/2.2.3/gems/jquery-rails-4.0.4", "file-digest///Users/schneems/app/assets/javascripts/foo.js"}>

To see if either of these has changed, Sprockets will pull their digests from the cache like we did with our first application.js asset. These are used to "resolve" an asset. If the resolved assets (and their dependencies) have been previously loaded and stored in the cache, then we can pull our asset from cache:

# https://github.com/rails/sprockets/blob/9ca80fe00971d45ccfacb6414c73d5ffad96275f/lib/sprockets/loader.rb#L55-L58

digest = DigestUtils.digest(resolve_dependencies(paths))
if uri_from_cache = cache.get(unloaded.digest_key(digest), true)
  asset_from_cache(UnloadedAsset.new(uri_from_cache, self).asset_key)
end

But now, our dependencies contain the full path. To fix this, we have to "compress" any absolute paths, so that if they're relative to the root of our project we only store a relative path.

Of course, it's never that simple.

Absolute paths everywhere

In the last section I mentioned that we would get a file digest by resolving an asset from `"file-digest///Users/schneems/app/assets/javascripts/foo.js". That turns out to be a pretty involved process. It involves a bunch of other data from the cache, which as you guessed, can have absolute file paths. The short list includes: Asset filenames, asset URIs, load paths, and included paths, all of which we handled in Pull Request #101. But wait, we're not finished, the list goes on: Stubbed paths, link paths, required paths (not the same as dependencies), and sass dependencies, all of which we handled in Pull Request #109, phew.

The final solution? A pattern of "compressing" URIs and absolute paths, before they were added to the cache and "expanding" them to full paths as they're taken out. URITar was introduced to handle this compression/expansion logic.

All of this is available in Sprockets version 3.3+.

Portability for all

When tested with an example app, we saw virtually no change to the initial compile time (around 38 seconds). The second compile? 3 seconds. Roughly a 1,200% speed increase when using compiled assets and deploying using Capistrano or Heroku. Not bad.

Parts of the URITar class were not written with multiple filesystems in mind, notably Windows, which was fixed in Pull Request #125 and released in version 3.3.4. If you're going to write code that touches the filesystems of different operating systems, remember to use a portable interface.

Into the future

Sprockets was originally authored by one prolific programmer, Josh Peek. He's since stepped away from the project and has given maintainership to the Rails core team. Sprockets 4 is being worked on with support for source maps. If you're running a version of Sprockets 2.x you should try to upgrade to Sprockets 3.5+, as Sprockets 3 is intended to be an upgrade path to Sprockets 4. For help upgrading see the upgrade docs in the 3.x branch.

Sprockets version 3.0 beta was released in September 2014; it took nearly a year for a bug report to come in alerting maintainers to the problem. In addition to upgrading Sprockets, I invite you to open up issues at rails/sprockets and let us know about bugs in the latest released version of Sprockets. Without bug reports and example apps to reproduce problems, we can't make the library better.

This performance patch was much more involved than I could have imagined when I got started, but I'm very pleased with the results. I'm excited to see how this affects overall performance numbers at Heroku—hopefully you'll be able to see some pretty good speed increases.

Thanks for reading, now go and upgrade your sprockets.


Schneems writes code for Heroku and likes working on open source performance patches. You can find him on his personal site.

Do You Believe in Programming Magic?

Unlike pulling a rabbit out of a hat, “magic” in programming is often performed under the guise of productivity. In this post, we’ll look at what defines a magical programming experience for better or worse. If you were in Rails around 2007 you might be quick to describe it as “auto-magical,” and this was a good thing. Magic meant freedom from the tyranny of endless boilerplate, freedom from hours of tedious configuration, freedom to be productive.

Keep reading Programming Magic on Codeship