Schneems - Programming Practices, Performance, and Pedantry

It's dangerous to go alone, `pub` `mod` `use` this.rs

Wed, 14 Jun 2023 00:00:00 +0000

What exactly does use and mod do in Rust? And how exactly do I “require” that other file I just made? This article is what I wish I could have given a younger me.

This post starts with a fast IDE-centric tutorial requiring little prior knowledge. Good if you have a Rust project and want to figure out how to split up files. Afterward, I’ll dig into details so you can understand how to reason about file loading and modules in Rust.

Tip: If you enjoy this article, check out my book How to Open Source to help you transform from a coder to an open source contributor.

Tutorial: Naievely require/load-ing files in Rust

Skip this if: You’re the type of person who likes to see all the ingredients before you see the cake.

For this tutorial, I’ll be using VS Code and the rust analyzer extension.

With that software installed, create a new Rust project via the cargo new command:

$ cargo new file-practice
     Created binary (application) `file-practice` package
$ cd file-practice
$ exa --tree --git-ignore ./src
./src
└── main.rs

Note: The exa command isn’t required. I’m using it to show file hierarchy. You can install it on Mac via brew install exa.

In the src directory, there’s only one file, main.rs. Since some people think tutorial apps are a joke, let’s make an app that tells jokes.

Create a new file named joke.rs by running:

$ touch src/joke.rs
$ exa --tree --git-ignore ./src
./src
├── joke.rs
└── main.rs

Add a function to the file:

// src/joke.rs
fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
}

Then modify your src/main.rs file to use this code:

// src/main.rs
fn main() {
    println!("Hello, world!");
    want_to_hear_a_joke();
}

This code fails with an error:

$ cargo test
error[E0425]: cannot find function `want_to_hear_a_joke` in this scope
 --> src/main.rs:4:5
  |
4 |     want_to_hear_a_joke();
  |     ^^^^^^^^^^^^^^^^^^^ not found in this scope

For more information about this error, try `rustc --explain E0425`.

You cannot use the code in src/joke.rs code as src/main.rs cannot find it.

When you create a file in a Rust project and get an error that it cannot be found, navigate to the src/joke.rs file in your VS Code editor and hit CMD+. (command key and period on Mac, or control period on Windows). You’ll get a “quick fix” prompt asking if you want:

“insert mod joke;”
“insert pub mod joke;”

Your editor may look different than mine, but note the quick-fix menu right under my cursor:

If you hit enter, then your src/main.rs should now look like this:

// src/main.rs
mod joke;

fn main() {
    println!("Hello, world!");
    want_to_hear_a_joke();
}

Watch for changes with cargo-watch

Skip if: You already know how to use cargo-watch in the VS Code terminal

Run your tests on save in the vscode terminal with cargo watch. You can open a terminal by pressing CMD+SHIFT+P (command shift “p” on Mac or control shift p on Windows). Then type in “toggle terminal” and hit enter. This will bring up the terminal. Then, install cargo watch

$ cargo install cargo-watch

Now run the watch command in your terminal:

$ cargo watch -c -x test

This command tells cargo to watch the file system for changes on disk, then clear (-c) the window and execute tests (-x test). This will make iteration faster:

Back to the tutorial

Make sure cargo watch is running and save the src/main.rs file. Note that tests are failing since the program still cannot compile:

   Compiling file-practice v0.1.0 (/private/tmp/file-practice)
error[E0425]: cannot find function `want_to_hear_a_joke` in this scope
 --> src/main.rs:6:5
  |
6 |     want_to_hear_a_joke();
  |     ^^^^^^^^^^^^^^^^^^^ not found in this scope
  |
note: function `crate::joke::want_to_hear_a_joke` exists but is inaccessible
 --> src/joke.rs:2:1
  |
2 | fn want_to_hear_a_joke() {
  | ^^^^^^^^^^^^^^^^^^^^^^^^ not accessible

For more information about this error, try `rustc --explain E0425`.
error: could not compile `file-practice` due to previous error
[Finished running. Exit status: 101]

The error “exists but is inaccessible” is similar to what we saw before but with additional information. If you run that rustc command, it suggests:

$ rustc --explain E0425
...
If the item you are importing is not defined in some super-module of the current module must also be declared as public (e.g., `pub fn`).

The very last line gives us a great clue. We need to update the function to be public. Edit your src/joke.rs file to make the function public:

// src/joke.rs
pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
}

On save, it still fails, but we’ve got a different message (always a good thing):

error[E0425]: cannot find function `want_to_hear_a_joke` in this scope
 --> src/main.rs:6:5
  |
6 |     want_to_hear_a_joke();
  |     ^^^^^^^^^^^^^^^^^^^ not found in this scope
  |
help: consider importing this function
  |
2 | use crate::joke::want_to_hear_a_joke;
  |

For more information about this error, try `rustc --explain E0425`.
error: could not compile `file-practice` due to previous error
[Finished running. Exit status: 101]

It suggests “consider importing this function” by adding use crate::joke::want_to_hear_a_joke; to src/main.rs. Use the quick-fix menu CMD+. on the function in src/main.rs:

After accepting that option, src/main.rs now looks like this:

// src/main.rs
mod joke;
use crate::joke::want_to_hear_a_joke;

fn main() {
    println!("Hello, world!");
    want_to_hear_a_joke();
}

When I save the file, it compiles!

   Compiling file-practice v0.1.0 (/private/tmp/file-practice)
    Finished test [unoptimized + debuginfo] target(s) in 1.34s
     Running unittests src/main.rs (target/debug/deps/file_practice-22ac5cd70121e9ea)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

[Finished running. Exit status: 0]

To recap what happened here:

We created a new file, src/joke.rs.
To make it visible to src/main.rs, we needed to add a mod statement. We used VS Code CMD+. to generate the mod statement.
We then updated the function in src/joke.rs to be public (added a pub based on the compiler detail error message)
Finally, we needed to add a use into src/main.rs so our code could call it. This was suggested both by the compiler and VS Code.

You don’t have to memorize EVERYTHING required. All in all, our tools either did the work or gave us a strong hint as to what to do next. Start mapping if-this-error to then-that-fix behavior while learning use, mod, and file loading.

Tutorial: Importing nested files

To import code from a file in the src/ directory into src/main.rs, you must add a mod declaration to src/main.rs.

How do you add files in a different directory?

Let’s say we want to tell several kinds of jokes, so we split them into a directory and different files. Use the results of the first tutorial and add to it:

$ mkdir src/joke
$ touch src/joke/knock_knock.rs
$ touch src/joke/word_play.rs
$ exa --tree --git-ignore ./src
./src
├── joke
│  ├── knock_knock.rs
│  └── word_play.rs
├── joke.rs
└── main.rs

Now add some code that we can import. Write a joke into src/joke/knock_knock.rs:

// src/joke/knock_knock.rs
pub fn tank_knocks() {
    println!("Knock knock.");
    println!("Who's there?");
    println!("Tank");
    println!("Tank who?");
    println!("You're welcome!");
}

And another into src/joke/word_play.rs:

// src/joke/word_play.rs
pub fn cow_date() {
    println!("Where do cows go on a date");
    println!("To the Moooooo-vies");
}

With these files saved, use the CMD+. quick-fix menu, which will prompt you to insert a mod. Select the first option on both files and save both.

You might be surprised it didn’t modify src/main.rs like our first tutorial. Instead, the extension modified the file src/joke.rs with these additions:

// src/joke.rs
mod word_play;   // <== New mod
mod knock_knock; // <== New mod

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
}

Notice:

When you use mod in your crate root (src/main.rs), it references a file in the same directory (i.e., src/<name>.rs).
When you use mod in a non-root file (like src/joke.rs), it references a file in a directory with that name (i.e., src/joke/<name>.rs).

Modify src/joke.rs to use the contents of those modules now:

// src/joke.rs
mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  word_play::cow_date();      // <== Here
  knock_knock::tank_knocks(); // <== Here
}

These two lines implicitly use the module imported above to its own namespace. You can also make this explicit using the self keyword:

// src/joke.rs
mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  self::word_play::cow_date();      // <== Here
  self::knock_knock::tank_knocks(); // <== Here
}

You can save and run this code. Make sure you’re sitting down so you don’t end up rolling on the floor laughing.

To recap what happened here:

We created two files in the src/joke/ directory. Each file has one public function.
We added two mod statements to src/joke.rs (via CTRL+.).
We used those two new namespaces inside of src/joke.rs to call the two new functions.

Calling `cow_date` from main

The want_to_hear_a_joke function will output two jokes. Sometimes my kids think a joke is so funny that they want to hear it again. Let’s add cow_date again. This time put it directly in the main() {} function:

// src/main.rs
mod joke;
use crate::joke::want_to_hear_a_joke;

fn main() {
    println!("Hello, world!");
    want_to_hear_a_joke();
    cow_date(); // <== Here
}

When you save, you’ll get an error:

   Compiling file-practice v0.1.0 (/private/tmp/file-practice)
error[E0425]: cannot find function `cow_date` in this scope
 --> src/main.rs:8:5
  |
8 |     cow_date();
  |     ^^^^^^^^ not found in this scope
  |
note: function `crate::joke::word_play::cow_date` exists but is inaccessible
 --> src/joke/word_play.rs:2:1
  |
2 | pub fn cow_date() {
  | ^^^^^^^^^^^^^^^^^ not accessible

For more information about this error, try `rustc --explain E0425`.
error: could not compile `file-practice` due to previous error
[Finished running. Exit status: 101]

We’ve seen this error before “cannot find function <name> in this scope”. The help ends with this line:

“If the item you are importing is not defined in some super-module of the current module, then it must also be declared as public (e.g., pub fn).”

That’s not super helpful since the cow_date function is already public:

// src/joke/word_play.rs
pub fn cow_date() { // <== Note the `pub` here
    println!("Where do cows go on a date");
    println!("To the Moooooo-vies");
}

Before we saw that these two invocations are basically the same thing:

// src/joke.rs

self::word_play::cow_date(); // <== Explicit self
      word_play::cow_date(); // <== Implicit self

So you might guess that calling cow_date() inside of src/main.rs is the same as calling self::cow_date(), which makes it a bit more explicit. You might also notice that cow_date is nowhere in src/main.rs. Where did it come from?

That function came from src/joke/word_play.rs, but Rust cannot find it. The first time we called the function, we started with self and traversed the path. Let’s try that same technique:

// src/main.rs
mod joke;
use crate::joke::want_to_hear_a_joke;

fn main() {
    println!("Hello, world!");
    want_to_hear_a_joke();
    self::joke::word_play::cow_date(); // <== HERE
}

Did that work?

error[E0603]: module `word_play` is private
 --> src/main.rs:8:17
  |
8 |     self::joke::word_play::cow_date();
  |                 ^^^^^^^^^ private module
  |
note: the module `word_play` is defined here
 --> src/joke.rs:2:1
  |
2 | mod word_play;
  | ^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0603`.
error: could not compile `file-practice` due to previous error
[Finished running. Exit status: 101]

No. But, our message changed again (always worth celebrating). Before, the error said that cow_date was “inaccessible.” Now it’s saying that word_play is a private module.

It points at a line in src/joke.rs, where word_play is defined for self::joke::word_play.

Hover over word_play in src/main.rs and press CMD+.. It asks if you want to change the visibility of the module:

Accept that change and save. Then the file compiles!

To recap what happened here:

We added a function call to cow_date() in src/main.rs
Main.rs couldn’t find our code, so we used the full path self::joke::word_play::cow_date()
That gave us an error hinting the problem was privacy related.
We changed mod word_play; to pub(crate) mod word_play; in src/joke.rs (by using the quick fix menu CMD+.)
Now the program works

You might wonder, “What’s pub(crate) and how is it different from pub?

The pub(crate) declaration sets visibility to public but is limited to the crate’s scope. This is less privileged than pub. The pub declaration allows a different crate with access to your code (via FFI or a library) to use that code. By setting pub(crate), you indicate a semi-private state. Changing that code might affect code in other files of your project, but it won’t break other people’s code.

I prefer using pub(crate) by default and only elevating to pub as needed. However, the core part of this exercise was seeing how far we could get by letting our tools figure out the problem for us.

How can I load that file?

If all you want to do is put code in a file and load it from another file, you’re good to go. This is the high-level cheatsheet for what we did above:

Reference Rust code in a file fast by:

Starting from the bottom of your file tree in the project, use CMD+. and insert mod statements.
Continue the first step until you’ve reached the root of your crate (src/main.rs or src/lib.rs).
Try to use the code where you want while making the best guess for the correct path (i.e., self::joke::word_play::cow_date)
Use compiler errors and the quick-fix tool (CMD+.) to insert use statements or adjust visibility with pub or pub(crate).

What is that `mod.rs` file?

If you use the above methodology, that rust analyzer will create files. There are two ways to load files from a directory. Before, I used the convention that src/joke.rs loaded all the files in the src/joke directory. That’s technically the preferred way of loading files in a directory, but there’s one other method which is: putting a mod.rs file in the directory you want to expose.

In short, src/joke/mod.rs would do the same thing as src/joke.rs. Please don’t take my word for it. Try it out:

Here’s the current file structure:

$ exa --tree --git-ignore ./src
./src
├── joke
│  ├── knock_knock.rs
│  └── word_play.rs
├── joke.rs
└── main.rs

Now move the joke file to src/joke/mod.rs:

$ mv src/joke.rs src/joke/mod.rs

Now here’s what our directory looks like

$ exa --tree --git-ignore ./src
./src
├── joke
│  ├── knock_knock.rs
│  ├── mod.rs
│  └── word_play.rs
└── main.rs

If you re-run tests. They still pass! As far as Rust is concerned, this code is identical to what we had before.

Information dump

Now that you’ve tasted our lovely IDE productivity cake. It’s time to learn about each of the ingredients. Here’s what we’ll cover:

Code paths in Rust
- self and (unqualified)
- Differences between self and (unqualified)
- crate
- super
Using use
- Renaming paths with use
- Glob imports with use
- Import an extension trait with use
A file state cheatsheet with pub, mod, use cheatsheet
Modules != files

Code paths in Rust

In the above example, we saw that we can reference code via a path starting with self like self::joke::word_play::cow_date. Here’s what you can start a code path within Rust:

self
(unqualified)
crate
super

Code paths starting with `self` and (unqualified)

Starting a path with self indicates that the code path is relative to the current module. Before we saw this code:

// src/joke.rs
mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  self::word_play::cow_date();      // <== Here
  self::knock_knock::tank_knocks(); // <== Here
}

In this case, self is inside the src/joke.rs module. This keyword is optional in this case. You can remove it, and the code will behave exactly the same. Most published libraries do not use ‘ self ‘ because it’s less typing to omit the word altogether. Most would write that code like this:

// src/joke.rs
mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  word_play::cow_date();      // <== Here
  knock_knock::tank_knocks(); // <== Here
}

In this code, self is implied because we’re not using crate or super.

Using self can be helpful as a reminder that you’re using a relative path when you’re starting. If you’re struggling to correct a path, try mentally substituting the module’s name (in this case, joke) for the self keyword as a litmus test to see if it still makes sense.

Differences between `self` and (unqualified)

While it might seem that self and unqualified are interchangeable, they are not. This code will compile without self:

// Compiles fine
fn main() {
    println!("Hello, world!");
}

However, this code will not:

// Fails to compile

fn main() {
    self::println!("Hello, world!");
}

With error:

error[E0433]: failed to resolve: could not find `println` in the crate root
 --> src/main.rs:6:11
  |
6 |     self::println!("Hello, world!");
  |           ^^^^^^^ could not find `println` in the crate root

In addition to being an implicit reference to self, using an unqualified path also allows you access to elements that ship with Rust, like the println! macro or std namespace:

fn main() {
    let _out = std::fs::read_to_string("Cargo.toml").unwrap();
}

If you put self:: in front of std::fs above, it would fail to compile.

Using an unqualified path also gives you access to any crates you import via Cargo.toml. For example, the regex crate:

$ cargo add regex

fn main() {
    let _re = regex::Regex::new("lol").unwrap();
}

If you put self:: in front of regex above, it would fail to compile.

Code paths starting with `crate`

A code path that starts with crate is like an absolute file path. This keyword maps to the crate root (src/main.rs or src/lib.rs).

In our above example, self was also crate since we were in the src/main.rs, so we could have written this code like this:

// src/main.rs
self::joke::word_play::cow_date();

As this:

// src/main.rs
crate::joke::word_play::cow_date();

Because self refers to src/main.rs, these two lines of code are identical. However, if you try to copy and paste them to another file, only the one that starts from crate will continue to work.

You can use an absolute path in any code module as long as all the parts of that path are visible to the current module. For example, here’s the src/joke.rs file using a mix of absolute and relative paths:

// src/joke.rs
pub(crate) mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  crate::joke::word_play::cow_date(); // <== Absolute
  self::knock_knock::tank_knocks();   // <== Relative
}

This code calls cow_date via its absolute cargo path and tank_knocks from a relative path.

Code paths starting with `super`

The super path references the path of a parent. In this case, the super of src/joke.rs is src/ (otherwise known as the crate root). You can re-write the above code using super then as a replacement for crate like this:

// src/joke.rs
pub(crate) mod word_play;
mod knock_knock;

pub fn want_to_hear_a_joke() {
  println!("Want to hear a joke?");
  super::joke::word_play::cow_date(); // <== HERE
  self::knock_knock::tank_knocks();
}

Code path recap

self - A relative path from the current module
(unqualified) - Same as self but allows for std, crate calls, and more
crate - An absolute path from crate root (src/main.rs or `src/lib.rs).
super - Relative path to the parent of the current module.

Using `use`

Renaming paths with use
Glob imports with use
Import an extension trait with use

Renaming paths with `use`

Skip this if: You know how to use use to rename modules.

In Rust, there is a filesystem module std::fs, but you don’t have to use it to well…use it. You can type out the full path:

fn main() {
    let path = std::path::PathBuf::from("/tmp/lol.txt");
    let out = std::fs::read_to_string(path).unwrap(); // <== HERE
    println!("{out}")
}

Instead of writing out std::path::PathBuf and std::fs everywhere, you could tell Rust that you want to map a shorthand and rename it:

use std::path::PathBuf as PathBuf; // <== Here
use std::fs as fs;                 // <== Here

fn main() {
    let path = PathBuf::from("/tmp/lol.txt");
    let out = fs::read_to_string(path).unwrap();
    println!("{out}")
}

This code says, “Anytime you see PathBuf in this file, know what I mean is std::path::PathBuf”. This pattern is so common you don’t need the repetition of as at the end. Since you’re use-ing it as the same name, you can write this code:

use std::path::PathBuf; // <== Note there's no `as`
use std::fs;            // <== Note there's no `as`

fn main() {
    let path = PathBuf::from("/tmp/lol.txt");
    let out = fs::read_to_string(path).unwrap();
    println!("{out}")
}

All three programs are exactly the same. The use does not do anything. It simply renames things, usually for convenience.

Glob imports with `use`

Beyond importing a namespace or a single item (such as an enum, struct, or function), you can import ALL the items in a namespace using a glob import.

For example:

use std::path::*; // <== Note glob import
use std::fs::*;   // <== Note glob import

fn main() {
    let path = PathBuf::from("/tmp/lol.txt");
    let out = read_to_string(path).unwrap();
    println!("{out}")
}

In this code, we can call the read_to_string function without naming it above. That’s because read_to_string exists at std::fs::read_to_string, so when we import std::fs::*, it gets imported along with all of its std::fs::* friends.

This is generally discouraged because two imports may conflict. If one-day std::path introduces a new function named std::path::read_to_string, then there would be a conflict, and your code would fail to compile.

While glob imports might save time typing, they increase the mental load while reading. They’re not currently considered “idiomatic.” The exception would be for a prelude file within a crate.

Import an extension trait with `use`

Beyond renaming imports, use can change your code’s behavior!

In Rust, you can define a trait on a struct that wasn’t defined in the same library through something known as an “extension trait”. This might sound exotic and weird, but it’s common. Even Rust core code uses this feature.

For example, this code will not compile:

fn main() {
    let number = u64::from_str("99").unwrap();
    println!("{number}")
}

You’ll get this compile error:

   Compiling demo v0.1.0 (/private/tmp/demo)
error[E0599]: no function or associated item named `from_str` found for type `u64` in the current scope
 --> src/main.rs:2:23
  |
2 |     let number = u64::from_str("99").unwrap();
  |                       ^^^^^^^^ function or associated item not found in `u64`
  |
  = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:
  |
1 | use std::str::FromStr;
  |
help: there is an associated function with a similar name
  |
2 |     let number = u64::from_str_radix("99").unwrap();
  |                       ~~~~~~~~~~~~~~

For more information about this error, try `rustc --explain E0599`.
error: could not compile `demo` due to previous error
[Finished running. Exit status: 101]

But if you add a trait via use, now this code will compile:

use std::str::FromStr; // <== Here

fn main() {
    let number = u64::from_str("99").unwrap();
    println!("{number}")
}

Why did that work? In this case, the use statement on the first line changes the code by bringing the FromStr trait into scope. Also, Rust was clever enough to realize that it could compile if we added a std::str::FromStr onto the code. It’s right there in the suggestion. Neat!

This behavior change confused me for the longest time as most use documentation and tutorials just beat the “Rust does not have imports, only renaming” drum to the point that it’s not helpful information. Confusingly enough, if you define the FromStr trait on your own struct, you’ll have to also use std::str::FromStr everywhere you want to use that behavior too.

File/State permutations

The various permutations of pub, mod, and use can be confusing, to say the least. I wrote this out as an exercise in understanding them. It’s more useful as a reference:

Adding the code mod <name>;
- Inside of <filename>.rs will:
  - Allow access to contents of <filename>/<name>.rs via <name>::<component>.
  - Allow access to <filename>/<name>/mod.rs via <name>::<component>.
  - Allow access of submodules of <filename> to access <filename>::<name>.
- Inside of a module root (i.e., <dirname>/mod.rs)
  - Allow access to <dirname>/<name>.rs via <name>::<component>.
  - Allow access to <dirname>/<name>/mod.rs.rs via <name>::<component>.
  - Allow access of submodules of <dirname> to access <dirname>::<name>.
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Allow access to src/<name>.rs via <name>::<component>.
  - Allow access to src/<name>/mod.rs.rs via <name>::<component>.
  - Allow access to submodules of crate to access crate::<name>.
Adding the code pub(crate) mod <name>;
- Inside of <filename>.rs will:
  - Same as mod <name> plus.
  - Allow super to access <filename>::<name>.
- Inside of a module root (i.e. <dirname>/mod.rs).
  - Same as mod <name> plus.
  - Allow super to access <filename>::<name>>
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Same as mod <name>.
  - No additional effect.
Adding the code use <module>::<name>; (relative path)
- Inside of <filename>.rs will:
  - Add an alias from <module>::<name> as <name>.
  - Bring an extention trait into scope (if <module>::<name> is an extension trait).
  - Allow sub modules to access <filename>::<name> without using <module>.
- Inside of a module root (i.e., <dirname>/mod.rs)
  - Add an alias from <module>::<name> as <name>.
  - Bring an extention trait into scope (if <module>::<name> is an extension trait).
  - Allow sub modules to access <dirname>::<name> without using <module>.
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Add an alias from <module>::<name> as <name>.
  - Bring an extention trait into scope (if <module>::<name> is an extension trait).
  - Allow all modules to access crate::<name> without using <module>.
Adding the code pub(crate) use <module>::<name>; (relative path) will do everything as use <module>::<name>; plus:
- Inside of <filename>.rs
  - Same as use <module>::<name>; plus.
  - Allow super to access <module>::<name>; as <filename>::<name> without using <module>.
- Inside of a module root (i.e., <dirname>/mod.rs)
  - Same as use <module>::<name>; plus.
  - Allow super to access <module>::<name>; as <dirname>::<name> without using <module>.
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Same as use <module>::<name>;.
Adding the code use crate::<module-path>::<name> (absolute path)
- All modules in the path must be reachable with public visibility to crate root or there will be an error.
- Inside of <filename>.rs
  - Same as use <module>::<name>; but the full path must be reachable from crate root.
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Same as use <module>::<name>; but the full path must be reachable from crate root.
- Inside of a module root (i.e., <dirname>/mod.rs)
  - Same as use <module>::<name>; but the full path must be reachable from crate root.
Adding the code pub(crate) use crate::<module-path>::<name> (absolute path)
- Inside of <filename>.rs
  - Same as use crate::<module-path>::<name>;
  - Allow all modules to access crate::<path-to-filename>::<name> without referencing <module-path>.
- Inside of a module root (i.e., <dirname>/mod.rs)
  - Same as use crate::<module-path>::<name>;.
  - Allow all modules to access crate::<path-to-dirname>::<name> without referencing <module-path>.
- Inside of a crate root (i.e., main.rs, lib.rs)
  - Same as use crate::<module-path>::<name>;.

Modules != files

I’ve focused on mapping modules and files, but you can use modules without files. Rust by example modules gives some examples. You’ll likely find modules used without filenames as you go through various docs on use or the module system. Aside from mod.rs and <dirname>/<modulename>.rs, most features map 1:1 whether or not your modules are backed by a file system.

I really wanted to call this a “comprehensive” guide, but there’s more to this rabbit hole. If you want a depth-first kind of learner, you can dive into some further reading:

The Rust Reference (not THE book) - Talks about this stuff in detail
- Modules
- Use declarations
Rust edition guide talks about changes to the module system.

What is github.com/zombocom and why most of my Ruby libraries there?

Fri, 24 Feb 2023 00:00:00 +0000

The other day I got another question about the zombocom org on GitHub that prompted me to write this post. This org, github.com/zombocom, holds most all of my popular libraries). Why put them in a custom GitHub org, and why name it zombocom? Let’s find out.

Why a custom org?

If you’re maintaining one or two libraries, keeping them in your GitHub user’s namespace is easy enough. For me, this is https://github.com/schneems. The zombocom org has 18 libraries, 16 of which I created.

I want to encourage people to contribute to my libraries, so I’ve taken to giving commit access to developers who land a successful PR. I still control releasing, so this permissive default is intended to allow developers with ambition to express themselves while giving me an opportunity to QA and chime in on changes.

I wanted to take this a step further and give them access to ALL my libraries to make it even easier to contribute. This strategy is time-consuming if my libraries are all under github.com/schneems, so I moved them into a custom org name and called it a day.

Why name it zombocom?

Making a custom org name is familiar for top-rated gems/libraries. For example, github.com/puma or github.com/CodeTriage. My most popular library on zombocom is get_process_mem with 52 million downloads. But an org with that same name would be overly restrictive.

Custom orgs are usually named around a common theme. But my libraries don’t have a shared theme aside from being a thing I wrote that hopefully makes your life easier.

I didn’t want 16 namespaces for 16 libraries. I wanted a place where anything was possible. So I named it after Zombo.com:

Anything is possible at Zombocom. The infinite is possible at Zombocom. The unobtainable is unknown at Zombocom. Welcome to Zombocom

Another org inspired this move github.com/sparklemotion. This org was initially used for Nokogiri (the most popular XML/HTML parsing library for Ruby), and other libraries were added over time. The name is a joke from the movie Donnie Darko:

Sometimes I Doubt Your Commitment To Sparkle Motion.

The Nokogiri library and the sparklemotion org were created by Aaron Patterson, aka tenderlove. I asked him about it at a conference once, and he said he made it for similar reasons.

Some ask if I also made Zombo.com, but the answer is no. I have nothing to do with that site and don’t know who originally wrote it. The name is a tribute to a meme before memes. If the author ever wants to share my org name to put their website source, that would be fun. After all, anything is possible.

If you want to get started in open source, but need a helping hand, check out my (paid) book How to Open Source or (free) service CodeTriage

Pairing on Open Source

Wed, 09 Nov 2022 00:00:00 +0000

I came to love pairing after I hurt my hands and couldn’t type. I had to finish up the last 2 months of a graduate CS course without the ability to use a keyboard. I had never paired before but enlisted several other developers to type for me. After I got the hang of the workflow, I was surprised that even when coding in a language my pair had never written in (C or C++), they could spot bugs and problems as we went. Toward the end, I finished the assignments faster when I wasn’t touching the keyboard, than I was by myself. Talking aloud forced me to refine my thoughts before typing anything. It might be intimidating to try pairing for the first time, but as Ben puts “it’s just a way of working together.”

This Hacktoberfest, I started a Slack group for the ~350 early purchasers of my book How to Open Source. In the intake survey, they told me they wanted to learn more about pairing. When I think pairing, I think of Ben Orenstein, CEO of the pairing app tuple.app. I jumped on Twitter and asked if I could interview him for the group, and he agreed!

Listen in as we discuss the intersection of pairing and open source contribution. We’ll talk about how it’s different from regular pairing (or not), how to find people to pair with, and the best way to ask for help from a potential mentor.

In addition to talking about pairing in the group, we had developers who organized together to pair for Hacktoberfest. One made their first-ever contribution after their first pairing session. Then she wrote her first ever English blog post about the experience.

Topics

Is pairing on open source different?
How do you find people to pair with?
Can you match people in a group to pair?
FocusMate.com
Can developers of the same level pair?
What research has been done on pairing?
LearnToPair.com
How would you introduce pairing into a team?
tuple.app/oss - Free license for open source maintainers
Can you pair on non-technical tasks?
How do you initiate pairing sessions?

Transcript

(If you find a glaring problem with the transcript you can send me a PR to https://github.com/schneems/schneems.)

Ben: Yeah, let’s rock.

Schneems: Well, welcome everyone. My name is Richard Schneeman, author of How to Open Source. Today I have Ben, the ceo. CEO or CTO?

Ben: Ceo.

Schneems: Ceo. All right. Yeah, the big guns of tuple.app. So tuple is an application for pairing. It’s my favorite application for pairing coincidentally. So you wanna say Hi Ben?

Ben: Yeah, I’m stoked to be here. Pairing was a huge game changer in my career and so I’m stoked to talk about this topic that made a big difference for me professionally.

Schneems: Awesome, Awesome. Well I am very happy to have you. I recently launched a book and in doing so, asked as some of the people who bought it, what they’re interested in. A lot of ‘em indicated that they are really interested in pairing, so hence inviting Ben. And so yeah, wanted to ask you a couple of questions. One of ‘em that came up is just kind of high level, hey, are there differences pairing on open source software versus proprietary software?

Ben: Probably not in the actual act of pairing, I would assume. I can’t think of any differences that would be there. I think it’s probably, you might see some differences in what it takes to get people to pair with you. I’m not quite sure the willingness of open source maintainers to pair with other contributors. , you would sort of hope it was high, or maybe once you had earned a little bit of trust or shown a bit of promise, that could be a thing. That would be an easier pitch to make. If I were an open source maintainer and I sort had all the responsibility of my project, I would want to train other people on that project. So it's to help share some of the burden, I think. . And I haven't found a better thing than pairing for transmitting that sort of knowledge yet. So I would think, whereas if you're at a company and you're all are in the same team and you're all working on the same app, it's probably pretty easy to just DM somebody and say, Hey, can we pair on this thing?

Schneems: Totally, totally. Yeah. I think and you kind of alluded to a little bit of as a maintainer, needing to balance your like, yes, you wanna train people up, but you also need to balance that with, well hey, am I ever gonna see this person again? So in order to pair you to be able to find people, it’s a little bit different in working in the open, working in open source have you ever seen any sort of pairing groups or have you seen any sort of patterns with either a people who say like, Hey, I have a thing I wanna share, or other people come to the table and say, Oh, I have a problem, or I need help with this. Just, or tips really in general for people looking for someone to pair with, but maybe they don’t necessarily have a pre-built pool.

Ben: It’s kind of a bigger question. I think the question of how do you find someone to pair with is a little bit like the question of how do you find colleagues or mentors? Perry is just a way of working together. It’s not a magical practice. It’s effective and it’s a big fan of it, but it’s not different than working with other people really. It’s just a type of working together. And so I think it really transposes to the question of how do I find great people to work with? And I think maybe a short answer to that is be worth working with. Be a friendly person, a productive person. Show some indications that you are going to be good to pair with, is probably a good way to start . I think if you email someone and say, Hey, I'm looking for a mentor, will you be my mentor? No one wants to say yes to that. That's not a compelling pitch. But if you say, Hey, I've been a fan of your work. I loved this project and this project, you inspired me to do this thing and I ran into this problem, could you offer me, do you have any advice on how I might tackle this? Now you're starting a relationship with somebody. You've shown that you have done work, you've, you've done some homework already and you're showing that you are possibly worth mentoring. And I think that approach is probably likely to pay better dividends than saying just how do I find someone to pair with who wants to pair with me? Well, in the open source context, try to get a couple prs under your belt maybe or ask if the maintainers have considered a organizing a pairing day where maybe they connect a few people that are interested in becoming contributors or new contributors, something like that.

Schneems: , have you seen anything like that in the wild? And I ask because I, it's like, hey, I'm trying to run this community and it's like I've got people who want pair, I have people who wanna work on project. It seemed,

Ben: It’s a weird thing. I’ve seen a lot of failed efforts here. I think it’s a pretty common programmer impulse to be like, I should build a site that matches pair people together that wanna pair programming. Cuz you could sort envision how the Apple work. And so a lot of programmers write it and I think it’s not a matching problem really just because these two people wanna pair in the same time zone and in the same language doesn't mean they're really gonna actually pair together. I think there's more social complex social dynamics at play than an app is going to solve. I think the closest thing I've seen to a direct effort to cause more pairing that has worked well has been inside a company. So one of the customers of Tuol is Shopify. They have thousands of developers using our product and they ran a internal pairing contest. They ran a contest and it was like whoever pairs the most this week or with the most people or the most time becomes eligible for these prizes. And I think the prizes were honestly just t-shirts and badges and things like that. I think they were fairly simple things, But I think you, I remain continually surprised by how much developers will do for a . So , that might be a possible approach

Schneems: There. A hundred percent. Yeah. I mean we’re definitely seeing that there’s actually a lot of engagement within the group with within Oktoberfest is just something to focus on, I feel like is a big thing. When you said that, I was like, Oh yeah, it’s like a contest. It’s like, yeah, who can pair the best? It’s who can be the most friendly . Ah, I'm gonna crush it at being friendly.

Ben: Totally.

Schneems: That’s interesting. Yeah. Pairing contest. Yeah. And even one of the people in the group specifically called out and mentioned Focusmate. Have you ever heard of Focus Mate?

Ben: Yeah. That’s the thing where you go and you work with somebody kind of simultaneously, but separately,

Schneems: Right? Yeah. So for just everybody else, you log into the site and you just say, Hey, I need to focus. I need basically somebody to keep me accountable. It pairs you up with someone, I mean not pairs. It matches you with someone. There we go. And then you basically just sit there for a block of time. Well you both do work. And the idea is if you start playing on your phone or something, the other person would see. So it’s, it’s working in a cafe. So yeah. Good. You’re familiar with it. Somebody specifically was asking like, Hey, have you ever considered integrating with topple? And it, it’s back with the matching issue. Not super keen on that, like you said, as being a technology issue.

Ben: Yeah, I don’t believe that you’re gonna pair random strangers together and have it worked that well. You might occasionally have success there if you get people with similar values and goals, you have higher success rate I’m sure. I think you probably want to pair strangers. I think you need to have a human in the loop and very motivated people. Otherwise it's kind of pairing up workout buddies where if any either person flakes, then the other person is also can't work out. It's now we're just tied together. In this particular foot race, we have considered building some things within teams. I think woodwork is saying Richard joined the team a week ago and hasn't paired with anyone. Maybe someone should pair with him or here's a leaderboard. Richard actually pairs with the most people in a given month. I wonder if anyone could catch him. Here's the second and third and fourth place kind of thing. And sort of take a thing that is already mostly working or is already an established social group and kind of give it nudges. I think that would possibly work. But this we should add matching into twofold to have people find other people to pair with broadly throughout the internet and among strangers. I'm pretty skeptical of, I might be wrong, maybe it would be amazing feature, but I have suspicions,

Schneems: . Okay. Yeah, I makes sense.

Ben: Your group might be a good different, might be an exception. So we did do some. So when I was running Upcase for Thought bot, which is a educational developer training service I think we did do some matching of pairs and there was some success. I wouldn’t say it was resounding success, but there was pairing happening. People reported that it actually occurred. So I think it is possible within a group that has kind of self-selected and identified, I wanna learn this thing, I’m willing to invest this effort to do this

Schneems: for that. Within matching people. Of the times that you have mentioned pairing, it kind of sounds like you're mostly envisioning a different level. Senior, junior, I mean

Ben: No

Schneems: Actually no labels and whatnot or same level or I don’t know. Are there any other than values and character traits and just connection, are there any other things to look for that might make a good pair, a good pairing setup or a good pairing pair?

Ben: Pair programming is a more social endeavor than almost most programming activities. It’s live real time code review a little bit. And so just in a code coder view, you have to be a little careful about how you say things and you have to maybe use more emoji like positive emoji and more air on the side of politeness and friendliness and happiness than you might otherwise. . I think pairing has a little bit of that as well. I think a good pair is someone that has empathy and friendliness and a decent amount of easygoingness. And so I would seek to pair with people that you enjoy spending time with . If they're nice humans and pleasant and enjoy collaborating on things, then you'll probably have a good time pairing with them if they are the type to nerds snip you or while actually you all the time or be condescending. If you don't know something then you will not have a good time pairing with that person. And it's not parenting's fault, it just exposes a thing that was there.

Schneems: Right. Yeah. There’s just already a relationship. It’s not gonna be literally a different person when they show up to the

Ben: Yeah. Session.

Schneems: Yeah, that makes sense to me. So on the site you do have why pairing and there there’s sort a five second, five minute and maybe that’s not the delineation. One of the people in my group was asking if you know of any research in the world of pairing. So they were trying to bring pairing to their company and they’re basically just looking for more ammo, fire power like hey, we say pairing has done X, Y, and Z. Do you, you know of either a existing research or I don’t know, maybe ongoing research?

Ben: Yeah, if you Google scientific research into pair programming, our article is the top result I wrote. There’s not much is the short answer. So I wrote up what I found and it’s about six or so plausible studies that you could maybe say make a decent case for pairing . I think. So hopefully that will help. This is a site that we made learn to pair.com and that's one of the articles. There's lots of stuff there. Most of what I know about pairing, I tried to put in that site, learn

Schneems: To pair.com. Got

Ben: It. You try to justify this scientific, it feels like coming up with scientific research into programming productivity measures seems very hard to me. I think coming up with a measure of how productive programmers are is tantalizing and so far I think has been fairly intractable. I'm not sure you could get something that would convince a lot of expert practitioners that you have a good metric that actually demonstrates whether a programmer is productive. It might be almost impossible. And so to say here is some scientific research into the effectiveness of pair programming sort of requires some measurement otherwise you're just doing surveys. Did you feel more productive? Did you enjoy this experience more? And the surveys there, people have done this research with surveys and the surveys do show very strong positive results. People enjoyed pairing, they felt more productive, they felt more connected. I wouldn't probably take this track if I were trying to introduce pairing or try to justify pairing as an activity . I think I would try to no big deal my way into it. Meaning I think sometimes people think I need to convince my team, I should convince my team, my company, my management, that we should do pair programming. And so let me make the case. And it would be great if there were a study that showed that this is clearly a great idea. And I think what you probably should do instead is go to a coworker that you have a great relationship with and who is warm and friendly and empathetic and say, Hey, will you gimme a second set of eyes on this real quick and fire up something like Tuol or some other screen sharing tool or whatever. Or have them pull up a chair next to you. If you are working in person, plug in an extra keyboard and just write some code together. And don't even call it pairing. Don't make a big deal of it. Just be like, call it a second set of eyes. Cuz that's fundamentally what pairing is at the end of the day. It's two people looking at the same code at the same time. And all the rules or strategies or tactics about who types when and who has a mouse and who has a keyboard and where's the monitor And all this really is of implementation details, but the fundamental activity is two programmers looking at the same code. And so you could do that in person, you could do that remote and I would just casually start this practice and see how it goes. And if you like it and it works well try to do it some more and maybe do it with some other people. And maybe when someone at a standup says, Oh, I'm still stuck on that, whatever. And say, Oh, do you want me to come take a look at that with you at some point? And then lo and behold you're pairing with them or hey maybe, and Mary should pair on that thing or should work on that thing together. I would go at it grassroots and subtly and not make a big deal about it. And pairing is not for every team, so it might not work but I think that's probably gives you a pretty good chance of success. Top down, we will all pair can work occasionally. There are some organizations that have that and have done that and have become pairing organizations so it can work that's not how I would introduce pairing at an existing organization that is not already pairing or doesn't have extreme buy-in from the top who are willing to say, Yes, we're gonna make this. The priority of the quarter is everyone's gonna pair and we'll see how it goes

Schneems: . So I think the reason behind maybe why they were asking, it's like, Oh hey, how can I get this top down buy in? It's maybe they were looking for a credit card, they're looking for the sign in, they're looking for like, Oh hey, I it's specifically, it's like, oh, I heard there's this great app that I would like to get access to. Yeah. I don't know. It's like

Ben: What are their options?

Schneems: Yeah, yeah. Well it’s, You have a free trial period, right?

Ben: Yes, we do have a free trial. It doesn’t require a credit card card. So we see lots of our customers sign up as entry level developers or developers without a credit card. And then they try it and they like it and they ask someone high up the chain for permission to buy it and then become paid customers. But also and relevant to this group, I suspect, is that we give away the app for open source teams. So you can get a permanently free team if you are open source maintainers or working on open source and it’s just two.app/oss and fill out that form with what you’re working on. And we grant this to probably almost everyone that fills it out. So we leverage, we use a lot of open source software in our tool and our company is built on top of it. So we were very happy to give back to the OSS community in this way.

Schneems: And full disclosure, I am a recipient of this, a happy recipient of this program. And I think only so one, both people don’t need to have paid for, is that correct? Yeah. Okay. So it’s if you wanna pair with somebody who’s never done it, it’s they don’t have to make this big investment. If one person has a license, you can invite somebody else to pair with you.

Ben: Right? Exactly. Yep.

Schneems: Okay, cool. I think that helps ease that transition and gets to the heart of the, it’s like research maybe not, but it’s really, it’s how can we hit that ground running? How can we do more pairing? And I think to makes total sense to me. Have you ever heard of, I guess, pairing on non non-technical tasks or just almost like

Ben: Yeah,

Schneems: Writing an RFC or research or I guess you can speak to that.

Ben: I pair on non-technical tasks all the time every week. And this actually happens a lot at our company. A lot of non programming things happen on two sessions as a part of a pair. A lot of the benefit of pairing, there’s a lot of benefits of pairing that you get, even if there’s no code on the screen. So it’s often less boring to work on something or if something is boring, it’s less boring it’s less likely to be boring it's more engaging. If someone is there with you, it's harder to slack off if you're sharing your screen. So you tend to stay more focused and get the thing done faster with fewer Twitter accidental long Twitter breaks or email breaks or slack breaks or things like that. it spreads knowledge around the team. If I watch someone do a process or writing a doc or something, I'm learning that thing at the same time. So we are having all this rich communication happening. You can see how someone else works, how someone else thinks, which is really useful for just becoming more awesome yourself. You might see they have a wonderful Macs tool that you wanna steal or Oh hey, I didn't know you could do that in our app with this shortcut, or how did you make that thing happen through while programming, but also through while just using your computer. So there's all these benefits that are available to you. And I think it shines particularly well on coding because programming is so hard and it's just so easy to write bugs that having a second set of eyes is, and it's so hard to make a great design, a good coding design that it feels extra worth having another person there because bugs in production can be very costly. They're not fun to fix. They slow you down a lot. Bad design decisions probably that even worse. True for all those things, but even stronger probably . And so it shines a lot there. But we have found quite a lot of value in just pairing on running payroll or figuring out what's the policy on this thing or what are our goals for next quarter, all those sorts of

Schneems: Things. Okay, cool. How do you initiate those conversations? Or do you just say, Hey, I’m working on this, somebody wanted somebody wanna do it with me?

Ben: Yeah yeah. Same sort of casual way. I have standing pairing meetings, kinda like you my calendar with people that I work with a lot. True. That’s probably true for a number of our employees as well. But yeah, I think there’s also just a lot of, there’s a fair amount of ad hoc pairing happening where someone just call somebody else, I’m like, Oh, this is tricky. Let me just call this person. And just they get a notification that I’m calling and there’s my screen and we start talking and it happens just kind of fairly fluidly and we’ve built a cult. Our culture is sort of steeped in this as you’d imagine making a pairing app for a living. Great. There’s a lot of pairing happening. Everyone’s expecting it. No one is shocked when you wanna do it with them. So that does a lot of work there.

Schneems: Cool. Very cool. Well yeah, I really appreciate you coming on board and answering some of the open source communities questions. So yeah, thank you Ben from couple.app.

Ben: Yeah, my pleasure. For.

My book 'How to Open Source' launches today!

Mon, 26 Sep 2022 00:00:00 +0000

Today is the day. How to Open Source is now available for purchase at howtoopensource.dev.

Buy the Book

I shared a pre-release copy of the book with some community members, and here’s what they thought of the book:

“Richard is a rare voice in the open source community: skilled and experienced, but also empathetic, caring and welcoming. If you’re venturing into the open-source bazaar, let Richard be your guide.” - Nate Berkopec, Maintainer of Puma webserver

“Richard does an excellent job of balancing the why of contributing to open source, with the how. This book will help you overcome common challenges folks face when navigating open source projects. If you find yourself on the other side of the table in the future, it will serve as a guide for how to make your project more welcoming to others. This book may be intended for those who already know how to code but is a useful read for anyone with a keen interest in open source.” - Anna Tumadóttir, Chief Operating Officer at Creative Commons

“How To Open Source provides practical, real world examples of getting started in Open Source. This book is a must read for anyone that wants to get started in the open source world.” - Aaron Patterson, Ruby core contributor & cat owner

Read more about the book over at howtoopensource.dev.

Launch week is now over

Launch week was Sept 26 to Sept 30, 2022 and is now over. It was an amazing response. Thank you to everyone who bought! I’m sending out invites to the Slack channel and the discount codes have expired.

To hear about future promotions sign up for my mailing list below and follow @HowToOpenSource.

The room where it happens: How Rails gets made

Wed, 12 May 2021 00:00:00 +0000

Today I’m going to share my perspective on how Ruby on Rails is developed and governed and how I feel the Basecamp “incident” impacts the future of Rails. I’m going to start out telling you what I know for sure, dip into some unknowns, and dive into some hypotheticals for fun.

First off, who am I? Find me @schneems on Twitter and GitHub. I first contributed to Ruby on Rails ten years ago in 2011, and I’m in the top 50 contributors (by commits). I help maintain a few open source projects, including Puma and the Ruby Buildpack for Heroku (my day job). I’ve got 1,611,289,709 and counting gem downloads to my name.

In the Ruby on Rails ecosystem, I am known as a “contributor”. In Rails speak, that means that I also have commit access to the project.

Note “getting commit” means that the person can merge PRs, close issues, and (depending on configuration) push to the main branch.

The Basecamp incident

The reason I’m bringing this all up is that the Rails world has been reeling. DHH is not only the creator of Rails but also the Co-Founder of Basecamp. Basecamp has been in the news a lot after having ~1/3 of their employees quit. Many have also asked about Rails’ governance. To bring you up to speed on the loose timeline of the Basecamp news:

It’s hard to fully capture the response of the Twitterverse to the original announcements and news. A few voices of opposition were louder on my timeline than others and I wanted to share them for additional context:

Basecamp concerns hit Rails

To many, DHH is the face of Rails. DHH created Rails, he holds the Rails trademark, and he keynotes at RailsConf every year. When Basecamp news hit Twitter, concern became increasingly apparent with this Rails forum thread on Discuss that many people are worried about how the situation unfolding with Basecamp will affect Rails. The new word of the day was “governance”.

Basically, to most people, “how rails gets made” is a mystery box. That concern triggered Rails Core drafting and publishing a Rails Governance blog post which lists out who exactly is on Rails Core (helpful) and states that they operate through “consensus”.

Many people responded to the blog post. One that popped up on my timeline was Steve Klabnick’s “My “the project isn’t run by one person or company” T-shirt sure is raising a lot of questions answered by my shirt” tweet.

I interacted with that thread a bit, ultimately leading to this point where I’m now typing up a blog post.

How does someone get commit access to Rails?

I will start with how I got commit access. I wrote a script that emails me a Rails issue a day. I read the issues as they came in, learned from them, and eventually started commenting. My “issue triage” efforts were noticed, and I was invited to the “issue team”. In this process, I was granted commit to the project and given access to the #contributor Basecamp instance where chat happens.

I eventually turned that script into a service named CodeTriage

I’ve been around when others have been added as contributors. Generally, someone in the #contributors channel mentions they’ve noticed that a particular developer is helping a lot. They then suggest that we invite them to become a “contributor”. Usually, by that time, many in the #contributors’ room know this developer and agree to the addition. I don’t think I’ve ever seen someone be brought up for commit and get rejected, though I can’t guarantee it’s never happened. As far as I know, there’s not a formal process, and anyone in the #contributors channel could suggest/request someone else to be added.

In short, it comes down to this: A developer gets commit access to Rails after doing a significant amount of work on the project and is independently recognized or asked for commit access.

I don’t know if there are more formal criteria for getting access. I’ve been told that most activity happens in #contributors (as opposed to #core), and I’m inclined to believe there’s not much more to the process.

Rails Contributors vs. Rails Core

In addition to #contributors, developers can also become a member of “Rails Core”. While I took ownership of a significant Rails dependency (Sprockets, which frankly needs yet another new owner), I am not in Rails Core.

What does “Rails Core” mean if it doesn’t mean “commit”? This point is tricky and perhaps a bit subtle. It’s not written down anywhere. I’ve learned via trial and error and observation. One significant differentiator is that Rails Core members have release access to Rails itself. While I can cut a release of Sprockets, I do not have access to Rails/Railties/ActiveRecord, etc. The other big thing is seniority/authority/ownership.

Different Rails Core members seem to own different parts of the framework. Some more obviously than others. For instance, Xavier Noria previously owned the Rails autoloading code and now owns the Zeitwerk gem used by rails to replace the old autoloading code. Even though it’s not written down who-owns-what (that I know of), you can go into a part of the project and look at the commit history. If there is an owner of a specific file or sub-library, then usually, it’s not hard to pick out a recurring name.

In addition to ownership, there’s a seniority/authority aspect. I had a case where a Rails core member left me feedback on a documentation PR. They expressed disapproval of some of my decisions. I didn’t fully respond to the disapproval and merged the commit in myself. Later I got a private message from a Rails Core member (and Basecamp employee) saying that my behavior was not okay and that I shouldn’t go around/over/behind a decision from Rails Core. I apologized and listed actions I could take to help prevent it from happening again.

It makes sense that an official opinion from #core will over-write one from a #contributor, especially when the issue at hand is the contributor’s own work. I don’t make a habit of ignoring feedback, but also, at the time that I decided to merge my commit (I won’t try to justify it), I didn’t understand the severity of the action.

This dynamic of not knowing the full repercussions of administrative actions also exists outside Rails and open source. Recently at work, I submitted a PR to an internal project. The person who owns the project “approved” it, but it was a little unclear if they expected me to merge it or if they’re waiting for something else. In that case, I have their chat and know their working hours. I got the ambiguity resolved in 5 seconds by messaging them. In the fully async world of open source issues, a comment might take days to get a response (and sometimes never).

How I GUESS someone gets on Rails Core

While I’m not in “Rails Core,” I’ve seen many go into the ranks. I’ve also seen a number NOT become Rails core despite their massive involvement with the project. Based on those sets of people and my own experiences, I have some fan theories.

I’ve already mentioned some prereqs. A developer has to do a lot of work on Rails and already be recognized as a contributor. They’ve got to be willing to take over a large amount of code to maintain. They need to be active and present in their maintenance. I don’t know if this is a strict requirement or merely a coincidence, or a rite-of-passage, but it seems they must be willing to shepherd at least one Rails release.

From the outside-in, the most common cause for a prolific contributor to not be given the #core title seems to be disagreement with another core member. That being said, the only core member I know this has happened with for sure is DHH. I honestly don’t know if people are nominated, or they need to ask, or what. I’ve previously guessed that it would be possible for me to attain this “rank” if I could drop other open source commitments, be able to focus more time on Rails, and bring some “large” feature to the table. For instance, I’ve questioned/wondered if I could pitch making derailed_benchmarks a first-class Rails feature and if that would be a “large” enough project to merit a nomination. I’ve never asked publicly or privately, though, because I’ve not felt that I had the time or energy to take on the role fully. Would that work? I don’t know. I’ve not tried. But that’s my best guess.

Also, I want to take a moment to say a huge thanks to Rails Contributors but especially Rails Core…the amount of work they put into the project is enormous. While being on Rails Core is an honor, it’s also a significant commitment and a lot of work.

How do major features happen in Rails?

I view two sides of software - the tactical and strategic:

Tactical: Made or carried out with only a limited or immediate end in view (i.e., short-term goals)
Strategic: Of great importance within an integrated whole or to a planned effect (i.e., long-term goals)

All the tactical work of Rails happens out in the open, mainly on GitHub. Some parts of it happen in the #contributor room (but less than you might think). Strategic work is a bit harder to pin down.

My strategy for doing any work in Rails has been to break up any “strategic” initiatives into smaller individually comprehensible “tactical” initiatives. For example, in my blog post Container Ready Rails 5, I am pitching a larger feature/ability in Rails to manage a “containerized” Rails app better. In reality, this was a series of PRs made individually (which are linked in the post). While I had a strategy and this end goal, I didn’t start with a top-down design doc. I made PRs where I saw opportunities. I had conversations on GitHub issues, on Twitter, in Basecamp, and in person at conferences.

I was able to ship this project because I could break it down into smaller components. For more prominent features like Hotwire, Action Text, Action Storage, or DB sharding: the project is too big to squeeze in one-pr-at-a-time. In my view, there are two mainline ways to get a prominent feature that cannot be broken up into Rails.

The first way is to lay out the plans and get buy-in. This process might look like posting a design doc and asking for collaboration. The most recent time I’ve seen this happen is with a doc on “sharding” in Basecamp. I’ve never shipped something that large in Rails, but I’m guessing there’s also a decent amount of “whipping the votes” and getting buy-in. Even if it’s as tiny as mentioning “I wish we had feature X in Rails, don’t you?” at a conference.

The second path to getting a prominent feature in Rails is to extract it from Basecamp. There’s still a measure of buy-in and communication. There still may be plans posted, design docs, and the rest…but it’s a different tone in the conversation. When the Rails feature is coming as an extraction from Basecamp, it’s typically a conversation about “how” rather than “if”. It might be possible to get #core or #contributors to reject and block one of these extractions, but I’ve not seen it happen.

In the past, this strategy has served the community well as Rails ends up with features that have usually been semi-battle tested in production. It means that the implementers know the problem space well and are not designing features in a vacuum. Critics have noted that some features might be over-optimized for the Basecamp use case on the flip side. It’s a common enough joke that Rails is “Basecamp in a box”.

Explicit and Implicit Governance

Most of what I’ve laid out here is based on my observations. Not much about how “Rails gets made” is written down or official. This fact makes sense to me. While some Rails Core and Contributors work for companies that allow them to work on open source work hours, I don’t believe any of them are “full-time” on Rails. On the #core list, everyone is a programmer and a contributor first. Project management, people management, and coordination happen, but they’re emergent properties of the system.

What I like about this system: I’ve benefitted from this system. I’m at the top(ish) of that leaderboard. I have a great relationship with many/most other contributors. I love the Rails community. I think we’re a solid group. I’ve navigated this implicit system very well. I’ve taken risks to push the boundaries of what I’ve believed was acceptable action and discourse. I’ve also patiently built consensus very slowly for divisive changes over the years. The funny thing is that I didn’t even realize that’s what I was doing. I loved a system driven by programmers because it “made sense to me” about getting changes in. I loved that if I “did the work”, then I got the benefits. If there’s something in Rails I didn’t like, I felt empowered to change it.

What I don’t like about this system: Every step along the way, I’ve had to sit back, watch, and guess. This system fits my personality and style well. I’m used to finding ways to “work-steal” from others and find consensus. But there’s also plenty of times where I felt stuck. I listed one above. A PR was blocking other work, and I thought merging it wasn’t a big deal. I guessed wrong.

I feel confident working in this system, but I also feel afraid of working in this system. It can be exhausting to backchannel and “find buy-in” for every little thing. I don’t like that when I read the Basecamp news and had a visceral reaction, my first thought was, “Will my commit access be revoked if I share what’s on my mind?” It’s incredibly unclear what mechanisms exist to remove commit access from someone against their will, and also unclear what recourse those people can take to get it re-instated.

I also didn’t know what would happen to Rails, or what even could happen to Rails? While I’ve taken it for granted as a given that any Basecamp feature extractions are “guaranteed” (my opinion) to get into Rails, why? What could change that would make that not true anymore? As you’ve read this post, you’ve probably seen me mention the word “Basecamp” concerning how Rails works quite a bit. Would we continue to use it? Is it even possible not to use it? While these are specific questions that popped into my mind, the critical question is that I don’t even know if I can ask these questions without fear of retribution. Even hypothetically.

Moving forwards

Writing all this down made it clear to me that Rails is fundamentally built on politics. Every step of the journey involves implicit relationships that developers must navigate. The irony of the “no-politics” controversy is that it doesn’t remove the politics. It just removes the ability to talk about politics. It forces those private conversations to be public. Will there be repercussions for me writing this? I don’t know, and that’s a bit worrying.

What is my relationship with DHH? I’ve primarily done most of my work by flying under his radar. I’ve met him in person a few times. He’s always been personable, IRL. He’s never followed me on Twitter, and other than being involved in asking me to take ownership of Sprockets, I’m not sure he’s aware that I exist. I wish they had walked back their statements on Monday instead of doubling and tripling down, instead of going on network news to advocate others should do the same. I wish they came out strongly in favor of diversity and inclusion efforts. I wish they were vocal through their actions in dismantling white supremacy and in being anti-racist. Jason posted “an update”, which seems to be perhaps the beginning of some kind of acknowledgment. I think the damage has been done, and this event has highlighted deep, fundamental problems that cannot be so quickly whisked away.

DHH has not been in the Basecamp Rails #contributor chat since “the incident”. I don’t know what he plans on doing. I don’t speak for him. I don’t speak for anyone else in Rails Core or Contributors. These are my observations and opinions.

There’s a little activity in the contributor room, mostly about everyday getting-stuff-done topics. Some mentions of the forum post and chatter around it, but it’s quieter than I would have thought.

I want to use a framework called Nonviolent Communication (NVC) that has been valuable to my personal and professional life to try to be as straightforward as possible given the circumstances.

Observe: I wrote about the dynamics of how Rails is governed and the events that lead up to this post above. It’s not all pure “objective fact,” but instead, it’s what I’m seeing.

Emotion: I’m anxious over the future of Rails and the Rails community. I’ve got a fear that I should be doing something about that anxiety. I’ve also feared that if I take the wrong action, there will be some negative retribution. I’m feeling defensiveness in myself and the Ruby community. I’ve distinctly noticed that this topic hasn’t even come up at all on /r/ruby, even while it’s basically been the “twitter character of the day” on my feed for over a week and has regularly topped the orange site. I’m frustrated. I wrote this post in a partial attempt to process these feelings. wheel

Need: What I want is to build software as a community. I want to see from my community leaders that they can respect, value, and learn from the community. If a decision is made or action is taken that causes an uproar, I want reassurance that those voices have been heard. The ability to acknowledge a problem or disagreement is foundational to being able to work together.

I want to build software in a collaborative way where I can feel confident that I understand the impacts of my actions. I want to work with those that “Dare Greatly” (Brene Brown) and can let their guards down, be vulnerable, and (as a result) produce the best possible outcomes.

Request: This is where I narrow down a specific request to the listener so that my words won’t be misinterpreted (as much). It’s not a demand. It can be heard and acknowledged without the REQUIREMENT that it be acted upon. I ask you to open up about how you’re feeling and what you want to see happen. I see lots of bomb-throwing, I think it’s backed by a lot of rage, but I don’t know. I appreciate clarity from using NVC and might invite you to try on that framework. I don’t have a specific “fix” to recommend for this situation. Some have jumped to solutions. Some of those solutions may help. I’m not asking for a coup or trying to prevent one, for that matter. What I want is to be heard. I want clarity on the situation. I want to talk with more people and not feel entirely so alone in all of this.

Migrating a Ruby Library from TravisCI to CircleCI

Wed, 13 Jan 2021 00:00:00 +0000

TravisCI.org is dead. Long live the new CI! TravisCI.org was THE way to run CI for an open source Ruby library. It was so easy that it was seemingly effortless. Even better, it was free. Since the slow-motion collapse of the product, developers have been pushed to other CI providers. I was recently tasked with transferring CI away from Travis for my library derailed_benchmarks and chose CircleCI. This post is a little about why I chose CircleCI, a little about how the transition worked, and a little about nostalgia.

Nostalgia first

I vividly remember when I was working for Gowalla, we had no CI. I didn’t even KNOW what CI was until we got a new employee Brad Fults. Brad set up Jenkins on a mac mini, and slowly my world was transformed.

We didn’t have a strong testing culture when I started at Gowalla. I printed out a sign that read “ask me about Rspec” and taped it above my desk. We did have tests though. One issue that I had with testing is that everyone was expected to run tests locally before pushing to GitHub (or prod). But people wouldn’t, or they would forget. Sometimes it would work on their machine but break on someone else’s. One of my first OSS libraries was attempting to fix this problem I named it git_test. It would store the test results in git so you could see who was running tests and if they passed or failed.

After Brad set up the CI server, I didn’t love it at first. The CI machine was always down, or tests would pass locally but fail on CI. It took me longer than I would like to admit to realize that my ambition around creating git_test was solved by having CI and that CI was a far superior solution.

About the time Brad was setting up Jenkins, waves were being made about a hot new startup called TravisCI. It had a slick interface and unbelievably was all open source. I don’t remember if we ever transitioned over to using a CI provider at Gowalla. Still, I’ve come to rely on having a managed CI provider as an integral part of my software development practices over the years.

As Travis is shutting down their TravisCI.org, it feels like it’s the marker of an end of an era. It felt a golden age filled with promise and hope for collaboration between open source and private companies. The “free” offering felt less like a marketing gimmick and more like a bold declaration, that a company could make money, open source it’s own software, and support the community all at the same time. I mourn that.

Why CircleCI?

I now work for Heroku, and we have our CI product Heroku CI. I love it for application development. Heroku CI is not a great fit for testing libraries, especially since there’s no way to give global public access to the test output. In my day job, I use Heroku CI, and increasingly I’ve been using CircleCI for non-app testing. I like their tooling. You can use a local CLI for debugging and an easy “Run with SSH” option to debug in the cloud on their Hardware. That is a feature Travis never had. I’ve also interacted with their support several times and had great experiences.

Note: This is 100% my opinion. No one from Heroku has reviewed this post. This message is not endorsed by anyone other than me.

While I know many people are transitioning over to GitHub Actions for CI, I changed my Twitter avatar to “GitHub drop ICE” to protest their contract with the (known human rights abusing) ICE agency. I try not to use GitHub actions when I don’t have to. A less-political reason is performance.

I still use some actions like this one that checks if a PR touched the CHANGELOG.md. On this project, I also have CircleCI tests in Ruby, and frequently the CircleCI tests will boot, execute, and finish before the GitHub action even starts firing. While most of my test suites are dominated by the suite’s length, those extra seconds can add up.

Debugability and tooling are also on my mind. I can run a CircleCI test locally via the CLI, but cannot with GitHub Actions.

I also feel like CircleCI is a CI company while Github Actions is a feature tacked on to a git hosting service. It’s getting lots of love and resources now, but I don’t know ten years from now if that will still be true.

Transitioning from `.travis.yml` to `.circleci/config.yml`

Here’s my original .travis.yml for derailed_benchmarks:

language: ruby
rvm:
  - 2.2.10
  - 2.5.8
  - 2.7.1

gemfile:
  - gemfiles/rails_5_1.gemfile
  - gemfiles/rails_6_0.gemfile
  - gemfiles/rails_git.gemfile

jobs:
  allow_failures:
    - rvm: 2.2.10
      gemfile: gemfiles/rails_6_0.gemfile
    - rvm: 2.2.10
      gemfile: gemfiles/rails_git.gemfile

I love the compactness of this config. TravisCI was initially built with Ruby library maintainers in mind, and the compactness of the config in this case shows. It runs the tests (implicitly it knows to run rake test). It will run these tests against a “matrix” of Ruby versions (listed under rvm) and different gemfile contents (listed under gemfile). You can also see where I’ve configured it to skip some combinations that I know don’t work (Ruby 2.2 does not work with Rails 6.0).

Since this post is talking about config, you might want to view the latest derailed_benchmarks config. I won’t be keeping this post up-to-date with changes there, but the following may help you understand what the config is doing and how it fits together.

Now here’s what I ended up with for a roughly equivalent CircleCI config (with some Ruby versions changed):

version: 2.1
orbs:
  ruby: circleci/ruby@1.1.2
references:
  run_tests: &run_tests
    run:
      name: Run test suite
      command: bundle exec rake test
  # Needed because tests execute raw git commands
  set_git_config: &set_git_config
    run:
      name: Set Git config
      command: git config --global user.email "you@example.com"; git config --global user.name "Your Name"
  restore: &restore
    restore_cache:
      keys:
        - v1_bundler_deps-
  save: &save
    save_cache:
      paths:
        - ./vendor/bundle
      key: v1_bundler_deps- # CIRCLE_JOB e.g. "ruby-2.5"
  bundle: &bundle
    run:
      name: install dependencies
      command: |
        echo "export BUNDLE_JOBS=4" >> $BASH_ENV
        echo "export BUNDLE_RETRY=3" >> $BASH_ENV
        echo "export BUNDLE_PATH=$(pwd)/vendor/bundle" >> $BASH_ENV
        echo "export BUNDLE_GEMFILE=$(pwd)/gemfiles/$GEMFILE_NAME" >> $BASH_ENV
        source $BASH_ENV
        bundle install
        bundle update
        bundle clean

jobs:
  test:
    parameters:
      ruby_version:
        type: string
      gemfile:
        type: string
    docker:
      - image: "circleci/ruby:<< parameters.ruby_version >>"
    environment:
      GEMFILE_NAME: <<parameters.gemfile>>
    steps:
      - checkout
      - <<: *set_git_config
      - <<: *restore
      - <<: *bundle
      - <<: *run_tests
      - <<: *save

workflows:
  all-tests:
    jobs:
      - test:
          matrix:
            parameters:
              ruby_version: ["2.5", "2.7", "3.0"]
              gemfile: ["rails_5_2.gemfile", "rails_6_1.gemfile", "rails_git.gemfile"]
            exclude:
              - ruby_version: "3.0"
                gemfile: rails_5_2.gemfile

Break it down, start at the end

The most important part of the CircleCI config file is at the bottom. This is where I’m telling it about how to run my tests:

workflows:
  all-tests:
    jobs:
      - test:
          matrix:
            parameters:
              ruby_version: ["2.5", "2.7", "3.0"]
              gemfile: ["rails_5_2.gemfile", "rails_6_1.gemfile", "rails_git.gemfile"]
            exclude:
              - ruby_version: "3.0"
                gemfile: rails_5_2.gemfile

The workflows key is a special key, just like jobs and version. This last part says to define a workflow where it will run my job named test (defined above) with a test matrix that looks like this:

	rails_5_2.gemfile	rails_6_1.gemfile	rails_git.gemfile
Ruby 2.5	rails_5_2.gemfile-2.5	rails_6_1.gemfile-2.5	rails_git.gemfile-2.5
Ruby 2.7	rails_5_2.gemfile-2.6	rails_6_1.gemfile-2.6	rails_git.gemfile-2.6
Ruby 3.0	skip	rails_6_1.gemfile-3.0	rails_git.gemfile-3.0

Unlike TravisCI, Circle has no built-in understanding of Ruby or the gemfile, so we’ve got to define some lower-level primitives. In this case, we’re creating a parameter named ruby_version and another named gemfile, then the permutations of these two parameters will be used to build the test matrix.

You can also see that there’s a similar ability to “skip” combinations, though Circle uses the exclude keyword.

Consuming parameters

Now let’s look at the test job using the special keyword jobs:

jobs:
  test:
    parameters:
      ruby_version:
        type: string
      gemfile:
        type: string
    docker:
      - image: "circleci/ruby:<< parameters.ruby_version >>"
    environment:
      GEMFILE_NAME: <<parameters.gemfile>>
    steps:
      - checkout
      - <<: *set_git_config
      - <<: *restore
      - <<: *bundle
      - <<: *run_tests
      - <<: *save

First, the job declares that it accepts two parameters and should both be treated as a “string”.

    parameters:
      ruby_version:
        type: string
      gemfile:
        type: string

Next is the docker/machine declaration:

    docker:
      - image: "circleci/ruby:<< parameters.ruby_version >>"

CircleCI has a large number of pre-built docker instances that can be used. It doesn’t give me quite as much control as an rvm install via Travis, but since this is a library, I am mostly concerned with major and minor ruby versions. The parameters are substituted into this value using the << >> syntax:

Next I set an environment variable via my gemfile parameter to be used in a script:

    environment:
      GEMFILE_NAME: <<parameters.gemfile>>

The last thing in this section is the set of commands (or “steps”) that will execute for each test:

    steps:
      - checkout
      - <<: *set_git_config
      - <<: *restore
      - <<: *bundle
      - <<: *run_tests
      - <<: *save

Some steps are provided by CircleCI (like checkout), which checks out your source code, but you can either define yours inline like - run: "echo 'lol'" or you can use a reference as I’ve done here.

References

Each reference is defined with a name and a command. The restore and save references use some other pre-defined keys like restore_cache and save_cache.

references:
  run_tests: &run_tests
    run:
      name: Run test suite
      command: bundle exec rake test
  # Needed because tests execute raw git commands
  set_git_config: &set_git_config
    run:
      name: Set Git config
      command: git config --global user.email "you@example.com"; git config --global user.name "Your Name"
  restore: &restore
    restore_cache:
      keys:
        - v1_bundler_deps-
  save: &save
    save_cache:
      paths:
        - ./vendor/bundle
      key: v1_bundler_deps- # CIRCLE_JOB e.g. "ruby-2.5"
  bundle: &bundle
    run:
      name: install dependencies
      command: |
        echo "export BUNDLE_JOBS=4" >> $BASH_ENV
        echo "export BUNDLE_RETRY=3" >> $BASH_ENV
        echo "export BUNDLE_PATH=$(pwd)/vendor/bundle" >> $BASH_ENV
        echo "export BUNDLE_GEMFILE=$(pwd)/gemfiles/$GEMFILE_NAME" >> $BASH_ENV
        source $BASH_ENV
        bundle install
        bundle update
        bundle clean

Let’s look at this in step order:

    steps:
      - checkout
      - <<: *set_git_config
      - <<: *restore
      - <<: *bundle
      - <<: *run_tests
      - <<: *save

First is set_git_config:

  # Needed because tests execute raw git commands
  set_git_config: &set_git_config
    run:
      name: Set Git config
      command: git config --global user.email "you@example.com"; git config --global user.name "Your Name"

As the comment states, I only need this because some derailed tests are using raw git commands. When it executes, it will run git config --global user.email "you@example.com"; git config --global user.name "Your Name" on the shell.

Next is restore:

  restore: &restore
    restore_cache:
      keys:
        - v1_bundler_deps-

Here we’re setting a cache key based on the name of the CircleCI job. I want to store Gem dependencies in the cache so that test runs are faster. Setting this cache key is how it knows which cache to restore.

Once the cache is loaded I can install dependencies:

  bundle: &bundle
    run:
      name: install dependencies
      command: |
        echo "export BUNDLE_JOBS=4" >> $BASH_ENV
        echo "export BUNDLE_RETRY=3" >> $BASH_ENV
        echo "export BUNDLE_PATH=$(pwd)/vendor/bundle" >> $BASH_ENV
        echo "export BUNDLE_GEMFILE=$(pwd)/gemfiles/$GEMFILE_NAME" >> $BASH_ENV
        source $BASH_ENV

        bundle install
        bundle update
        bundle clean

A lot is going on here. I’m choosing to use bundler env vars instead of flags since some flags are deprecated. CircleCI needs to know about environment variable modifications between commands, so I’m writing to a file stored at $BASH_ENV that is sourced before every command.

Since derailed needs to test against many different Rails versions, it uses different Gemfile contents in the gemfiles/ folder. To set the correct one based on the parameters I used before, I am reading from the GEMFILE_NAME env var:

        echo "export BUNDLE_GEMFILE=$(PWD)/gemfiles/$GEMFILE_NAME" >> $BASH_ENV

I needed this value to be an absolute path since the tests execute sub-shells in different directories, so the $(PWD) here gets expanded to be an absolute path.

When all that config is written, then I source the file:

        source $BASH_ENV

Then I install dependencies via bundle install. Perhaps surprisingly, I then run a bundle update. Since derailed is a library, I don’t check in any Gemfile.lock since I don’t control the specific library versions that apps will use. Instead, this bundle update is telling bundler to check for more recent dependencies. I also need this because I’m caching dependencies and don’t want to get stuck on some ancient versions accidentally.

Finally, I execute bundle clean. That will remove any gem that’s not currently in the recently generated Gemfile.lock, which prevents the cache from getting bloated with many gems that we no longer need.

CircleCI does provide a Ruby “orb,” which is effectively some pre-packaged references that can be re-used. This concept is similar to GitHub action’s “marketplace”. The orb comes with some references to install a specific ruby version with RVM (which I don’t need since the docker container already has it). The orb can also be used for installing dependencies via bundler. Unfortunately, the Ruby orb is mainly optimized around application development and doesn’t expect things like the Gemfile to be in a different directory needed for library development. My method is verbose, but I feel pretty straightforward.

After dependencies are installed then they’re cached:

  save: &save
    save_cache:
      paths:
        - ./vendor/bundle
      key: v1_bundler_deps- # CIRCLE_JOB e.g. "ruby-2.5"

The last reference to mention is pretty self-explanatory:

  run_tests: &run_tests
    run:
      name: Run test suite
      command: bundle exec rake test

It runs my tests via bundle exec rake test. After all that, then my matrix of tests is up-and-running ship-shape.

Version and orbs

The last thing to mention is the version and orb declaration:

version: 2.1
orbs:
  ruby: circleci/ruby@1.1.2

The version, I believe, refers to the version of YAML API that you’re using. You can validate your YAML using the CLI $ circleci config validate.

I previously mentioned orbs. I use them in other places to install dependencies, such as pack for building Cloud Native docker images. I initially thought that I might want to have more control over my specific Ruby version (such as Ruby 2.7.2 instead of whatever comes on circleci/ruby:2.7). However, I ended up not needing that flexibility. I kept the orb here to have an excuse to talk a bit more about orbs, though, because if you end up using CircleCI, you’ll end up using orbs.

Here is an example of using the ruby orb via ruby/install-deps reference they provide in their example Rails config.

Retro

Honestly, this took a lot longer than I thought it would. I’ve not seen other people talk about using CircleCI for testing libraries, so I wanted to see what the result would look like. Overall I’m happy with the config results. Now that I’ve got the skeleton in place, making changes seems very easy, but it was a process to get here. I’m curious if anyone else uses CircleCI to test a library with multiple Ruby versions and multiple Gem files. If so, shoot me a link on Twitter @schneems.

Here’s some other examples of CircleCI test config for libraries:

Squash Unexpected-End errors with syntax_search

Tue, 01 Dec 2020 00:00:00 +0000

Have you ever hit an error that you just plain hate? Back in 2006, I was learning to program Ruby and following an example from a book. I typed in what I saw, hit enter, and ran into a supremely frustrating error message:

Array(values).map |x|
  x.upcase
end

# =>  syntax error, unexpected `end', expecting end-of-input

I was beyond confused about this unexpected end error. end is a keyword. Right? How could Ruby not expect end? After staring at the code for a while, I realized my mistake. My map |x| line should be map do |x| (missing the do). That moment was the start of a 14-year long hatred of that error message and a search for something better. This post is about how I wrote a gem that you can use to improve this error message. Keep reading!

Syntax Search: Extra end

Update: I changed the name of the lib from “syntax_search” to “dead_end”. All the explanations about how the code work still hold up.

TLDR;

To get improved “unexpected end” syntax errors in your project, add this to your Gemfile:

gem "dead_end"

Then make sure it’s required in your code:

Bundle.require # If you're using Rails, this is the default.

require 'dead_end'

If you’re using rspec, add this line to your .rspec file:

--require dead_end

This is needed because bundle exec rspec path/to/file.rb will throw a syntax error before the gem gets loaded

Now when you run your code, you get a helpful error:

SyntaxSearch: Unmatched `end` detected

This code has an unmatched `end`. Ensure that all `end` lines
in your code have a matching syntax keyword  (`def`,  `do`, etc.),
and that you don't have any extra `end` lines.

file: ~/spec/unit/env_proxy_spec.rb
simplified:

        1  # frozen_string_literal: true
        2
        3  require_relative "../spec_helper.rb"
        4
        5  module HerokuBuildpackRuby
        6    RSpec.describe "env proxy" do
    ❯   7      before(:all)
    ❯  10      end
       11
       # ...
      246    end
      247  end

Back Story

After that initial experience with this error, I mostly trained my brain that “unexpected end” (for me) means that I’m missing a “do” somewhere. That’s where most programmers stop. The error becomes background noise in their programming life. I’ve seen developers develop defense mechanisms for syntax errors like this. For example: making sure to run their code frequently, or making sure to have small commits so they can revert to working code quickly, or they scan the git diff. Either way, you slice it, the error message ultimately is not that helpful. Programmers adopt these practices as a work-around for this message. It’s like if there was a hole in your house, and instead of covering it up, you just spent time walking around it.

I remember vividly around 2013 I was working for Heroku, and we had Matz, Nobu, and Koichi from the Ruby core team fly into the office. We recorded an audio interview with them, and on the way out, I told Koichi and Nobu about this pain point. They listened, and we ultimately concluded that it was a difficult, if not impossible, problem to solve. I shelved the idea.

The impossible goal

Why is it so difficult to improve this error message? Here’s an example of code with a missing do:

it "touches a file" do
  Dir.chdir("/tmp")
    FileUtils.touch("myfile")
  end
end

This code has a syntax error, the Dir.chdir("/tmp") is missing a do. BUT that’s not what triggers the error. The parser tries to match each end to a corresponding syntax keyword (begin/def/do/if/while/etc.).

When Ruby tries to parse this code. It mistakenly thinks that the first end belongs to the first do:

it "touches a file" do # <== Here
  # Dir.chdir("/tmp")
  #   FileUtils.touch("myfile")
  end # <== Here
end

It then keeps parsing, and it tries to match the last end, but it can’t. The parser hits the end of the file which is unexpected. Then an error is raised.

In this example, it might be obvious to a human where the missing syntax was, but a computer cannot know the programmer’s intent. In this case, Dir.chdir() by itself without a block is VALID ruby code and will change the directory. Imagine if you found this code:

it "touches a file" do
  Dir.chdir("/tmp")
  Dir.chdir("foo")
    FileUtils.touch("myfile")
  end
end

One of these is expected to be a chdir with a block, but not the other. Without more information, a computer can’t KNOW which line is supposed to have the do. But you, my human friend, likely have a pretty good idea because there is extra information: the indentation.

Relax constraints and add information: Indentation informed syntax

Since it’s impossible to prove which line was intended to have the syntax, I relaxed my goals from “tell the user the problem” to “narrow the search space for the user to likely locations with a problem”. Finding the line that caused a syntax error isn’t impossible, which is why this blog post didn’t just stop above.

Humans use indentation to inform our decisions about what the author of the code intended it to do. Why not cut out some of the middle work and have the program narrow our search for us? Some IDEs will warn you if you have an end that does not have a corresponding syntax keyword on the same indentation. For example, my vim setup:

Originally I had the idea I could use that same detection idea, but I wanted to show the offending line, and not just the most likely end causing the problem. Beyond that, I wanted it to be robust for when indentation was a little off. By definition, when a syntax error occurs, the programmer has done something wrong. Only giving them a decent error when they’ve done everything else perfectly is not ideal. Also, I wanted to boost the signal to noise. Was it possible to remove code that I was pretty sure didn’t contain the error?

With all this in mind, I began to hunt for a solution.

Search for syntax

When you’re trying to drive from one place to another, you’ll find many ways to reach the destination. The way that Google maps works is it searches for a solution and has a heuristic for success. When it finds a solution that provably minimizes the heuristic (such as time to destination), it returns this to the user.

Usually, when you’re routing somewhere, you don’t want to be routed into a dead end. But what if your plan already had a dead-end in it? If you can remove a part of the plan and end up at your destination, then you know you’ve removed the dead end.

I view syntax error like a dead end in code. If we can remove a dead end from our route then we have a valid path. Another way to put this: if you remove a syntax error from a document, then the document becomes valid (it can be parsed). I used this concept to “search” for one or more syntax errors in a document.

Here’s a video of the process:

Syntax Search: Def missing an end

Defining the roads

When navigating between two locations in a car, there are natural constraints for how it moves. It’s constrained by gas and brake pedals, engaged gear direction, and steering wheel angle at the low level. But google maps does not need to go into that much detail. Instead, it just looks at the intersections.

Code has similar low-level constraints. Programs are made of individual characters but assembled into larger grammars that span multiple lines and eventually make an entire source file on disk. To search for a dead-end in the code, we need to find the right granularity. I chose to have my smallest unit be a line of source code and chunk the code into multiple lines to form “code blocks”.

Once we have one or more code lines, we can remove it to see if we’ve found our solution. We can also independently verify if that block can be parsed or not.

Like how Google maps mimics how a driver would behave, we can mimic how a human would begin to break apart logically and chunk source code. For example:

class Cat
  def eat

  def speak      # <==
    puts "meow"  # <==
  end            # <==
end

In this case, you might be able to see that the def speak “block” is valid. We can run it through a parser to confirm this programmatically. If the code block is valid, it cannot contain the syntax error, so it can effectively be commented out. My algorithm reduces it to:

  1  class Cat
❯ 2    def eat
  7  end

Once the def speak block was commented out, it then checked the def eat line and saw the document was valid without it. The code then returned. We’ve found the syntax error!

Turn-by-turn

The bulk of the heavy lifting is done in the code block generation. It starts at the furthest most indentation. I will then expand outwards until it hits a change in indentation. We parse all the internal code with the same indentation before “expanding” a block to a lower indentation level. As it works the program also marks and comments-out code that it’s found to be valid. It’s easier to visualize with an animation:

Syntax Search: Missing do

This process simplifies the code as it runs and has a high likelihood of producing a code block that contains the syntax error(s).

Complicating scenarios

Since I started this project knowing that the actual goal to “show you what line is missing syntax” is impossible, I’m left knowing that I’ll have to live with some unsatisfactory results. They’re imperfect because the intent is unclear.

Here’s an example:

class Cat
  def eat
    puts "nomnom"

  def speak
    puts "meow"
  end
end

Without involving a human, there could be two possible errors here. Maybe def eat is missing an end, but it’s also possible that the def speak line was supposed to be removed and the contents consolidated into one single block. Without more information, we cannot know. When my program tries simplifying this, it spits out:

  1  class Cat
❯ 2    def eat
❯ 5    def speak
❯ 7    end
  8  end

You can see a little more clearly that we could make this code valid by either removing line two or five or adding an end after line 2. That ambiguity is important and can be left up to the programmer.

Next steps

Try it out! Seriously. Syntax errors in the wild might look different than in the “lab” here. I need you, and more importantly, I need your syntax errors.

Installation instructions are found at the readme: install syntax_search in your codebase today!

Triage with Me - 11 issues & 2 PRs in 1.5 hours

Tue, 22 Sep 2020 00:00:00 +0000

Contributing to open-source can be intimidating, especially when you’re getting started. In this post and video series, join me as I triage 11 issues on a repo that I didn’t create and don’t have much experience with.

Update: For additional tips on creating and responding to issues, check out the How to Open Source book.

I didn’t edit the videos, so any mistakes and accomplishments are raw and live to see. I re-watched the videos and took notes on the types of problems I encountered and the questions and solutions that I explored along the way. I cleaned up my notes and put them all here along with the videos so you can follow along. Afterward, if you want to come back to a piece of advice or technique, you can skim my notes instead of re-watching the whole video.

While I don’t recommend you sit down and try to respond to every open issue in the repository, hopefully, by watching me triage issues, you can help get a sense of how you might be able to dig in and start contributing. As you’re watching, try asking yourself how you would respond and what questions you might ask.

The videos are split up into 4 sections with a bonus video at the end of my work on a PR to remove some deprecations. This open-source repo is written in Ruby, but the issues tended to be about higher-level concepts such as stdout/stderr and accessing APIs via HTTP. As a result, they should be accessible to someone from any programming language background. I’ve never triaged the issues on this repo before, so be prepared for a raw and real-time triaging session.

Recommended: Increasing your resolution on YouTube to the maximum so you can read the text better. Recommended: Increasing the playback speed to 1.5x or higher since I didn’t cut out the “umms” and pauses either.

If you like this “with me” series, find me on twitter @schneems and pitch me what you would like for me to work on live and record for another session.

Triage session 1/4

Issue 1 Heroics should use keep-alive connections #16

Use the GitHub UI to identify commentators of note Contributor/Author/Member
Ask: “Do I understand the issue?”
- List your assumptions, ask for clarification if needed
Ask: “Is this still a priority?”
- If we don’t know if it’s a priority, how could we find out? Is there a way for us to test our assumptions about prioritization? i.e., how could we write a benchmark to quantify “give some nice speedups.”
View: Code that’s mentioned Link#run
- Grab some example code or write some pseudo-code to validate we understand each other correctly.
- Read the docs of the mentioned code
Ask: “Can you point me at another location where this is implemented?”
Thoughts:
- As the triager, you do not need to fully understand every aspect of the conversation, but you need to work to drive the issue to completion.
- When someone talks about performance, ask for benchmarks or other ways we can quantify it. If they give benchmarks, run them, and verify the results.
- Context building is an important activity. If you don’t have enough context from the issue, likely, others don’t have enough either.

Issue 2 Examples produced by the CLI don’t show correctly help options #28

Click: Check linked issues for possible additional context.
View: If the reporter mentions code you’ve never seen before, find it.
- In addition to the code referenced, try to see if there are other references to the code. What is the context around that code?
Reproduce: On bug reports with reproduction steps, see if you can follow the reproduction steps and verify the bug.
- In the video, this is where we found the bug “uninitialized constant Moneta.”
- Ask: “Where is this code or constant defined.”
  - Search for where Moneta is defined (Is it internal? Is it external?)
- Moneta is defined as a dependency in the Gemspec
- Add the missing require to fix the failure
- The reproduction example runs now that the original reporter gave us without an error
Document: Deprecation warnings and other problems that come up that might be good for future contributions
Thought: Many issues straddle the line between feature and bug report. Bug reports are easier to work with than feature proposals. See if you can treat the issue as a bug.
When I tried to reproduce their example, I couldn’t, and it looks like the issue had already been fixed.
Thought: It takes work to verify a reproduction is valid or not, and the process can yield a lot of context (in this case, a PR to fix the moneta require issue even).
In writing out a response, state what you’ve observed and then what you believe to be accurate based on those observations.
- Explicitly state what you think the next steps should be (i.e., close the issue, more research, asked for clarification, etc.) and support the statement.
We took the fix for the moneta failure we saw earlier and made it into a PR.
Tool: I use the open-source GUI gitx to stage commits and write messages. It’s not required, though.
Ask: When making a PR, “Does this need a changelog?” “Does this need tests?”, or “do tests need to be updated?”

Triage session 2/4

Issue 3 feature: automatic generated web admin ui #36

For feature requests: What’s the context, opportunity, and implementation?
- Context: Looking for an easier way for non-programmers to interact with an API
- Opportunity: This could be solved by a web interface
- Implementation: They’re asking for this feature to be baked into this library, but no other details.
Try to find empathy and understanding. What drove the original creator to ask the issue?
Ask: For a feature request, is the best place for this feature inside of this library
- Can the feature be provided externally, via another library, plugin, or extension?
I say “sorry” not because I’m personally sorry in this case. However, instead, I empathize that the experience of getting no response is not great and that I still value their participation in the open-source community.
When suggesting issues should be closed, I suggest ways the issue creator can continue even though the issue is being closed. For example: porting the suggested feature to its own library.
Even closing issues requires empathy when doing it respectfully, and that empathy takes energy. The more people are helping, the more energy to go around.

Issue 4 Errors should go to stderr #40

I start off trying to do an ad-hoc demonstration of stdout and stderr
- Deprecation warnings go to stderr so that captured and piped output can be sent to other sources such as jq.
- To skip past me explaining what stdout and stderr are: jump to 14:46
The issue included a script to run that shows the problem. I try to reproduce the bug.
- Verify stated behavior is the actual behavior
- In the process of reproducing this stated bad behavior, I found another issue where the exit code $? returns a success status code (i.e. 0) even when the command failed due to an incorrect flag.
The report referenced a command heroics-generate. Unsure of its use, I looked it up in the project and found documentation on the README.md
- I see the issue reported is using a suggested pattern from the README
- While the issue states what they desire “to use stderr instead of stdout,” I see another way that could be used to address the issue using CLI arguments
I opened a new issue with the exit code behavior I saw before Issues need to describe the problem and describe what you expected to happen and what actually happened. Also, add relevant details like version numbers, operating system, etc. where relevant.

Triage session 3/4

Issue 5 Escape identifiers #41

Review linked resources
- Click “y” on a GitHub page to get a URL with the commit SHA in it. When you post the link back to the issue, it should always stay valid and point to the code at that point in time, even if the file is deleted or modified later.
- If someone links to a resource that is no longer valid, it’s helpful to find what they’re talking about and comment back with a link to it.
- Consider copying relevant information from linked resources (in addition to linking) so people can get context without having to move away from the issue.
You might notice that I use block quotes to reference things someone else has said and then reply to them. It’s unnecessary, but I find this convention can cut the confusion, especially if you find yourself wanting to respond to multiple things.
Ask: Is it possible that someone is using/abusing this bug as a feature
- If so, ask how likely that use is, what’s the impact to this person of fixing the bug, and if there’s any way to safely detect this usage so it can be warned/errored/deprecated.
- Even if you don’t see a “valid” reason why someone might use something non-standard, breaking it is still a breaking change. If you value stability for your project, you need to consider the migration experience. Keyword: Backwards compatibility.
If the issue contains a bug, ask for a reproduction or try to reproduce it.
- The reporter might be describing behavior that has been fixed. They might be using an older version. Ask for versions.
I use Grammarly for help with spelling and grammar. Reading sentences with grammar errors makes it harder to understand the underlying message. Taking time to clean up and clarify your comment means less time for other people reading.
- This is also true in any distributed communication environment, such as a remote job where you’re communicating via slack/email/issues.

Issue 6 chunked encoding support #42

I didn’t explain it, but a chunked response is when you send parts of your response back at a time, imagine 1/4 of a web-page sent four times, instead of all of the data in a single request.
When you can treat an issue as a bug report, it becomes easier to work with.
If the reporter only gives the expected response, ask for the current response/behavior.
Ask for a way to reproduce the issue.
- I link to https://www.codetriage.com/example_app to give the poster more context over what I’m asking for. Previously I would say, “can you give me a reproduction” and people would respond with several things from “what does that mean” to “my app is private, I can’t give you the source code.” This article intends to pre-emptively answer as many of these common questions as possible to guide the poster to give you what you need to drive the issue to completion.

Issue 7 Validating request payload client side #51

The more context required to understand an issue that isn’t explicitly laid out, the harder it will be to understand the problem, and the less likely progress will be made on the issue.
- The feature request is vague enough that I felt I had to spend time explaining it in the video. I could have asked for clarification instead.
Ask: Is the behavior the reporter is describing true?
- Gonna sound like a broken record: Ask for a reproduction
Read what other commenters have said, try to understand their comments.
A commenter left links to files that no longer are valid
- Find the correct links and get a “y” version (link to a specific commit).
Ask: If the proposed behavior is implemented, what are the consequences? Both good and bad. What if it’s not implemented? Is the bad that bad? Is the good that good?
Ask: Is there another way to achieve the requested goals?

Triage session 4/4

Issue 8 [Add `connect_url` or similar method to generated clients #52

](https://github.com/interagent/heroics/issues/52)

Read the issue, read the comments. Understand the feature request.
Consider the age of the feature request and the movement/momentum on the issue. Even if something is a good idea, it might not make sense to prioritize it.
This was the fastest issue that I closed so far. I mention that it might be I’m running out of emotional energy (i.e., patience). You need to be protective of your energy and time, but also you don’t want to do a sub-par job.

Issue 9 Exposed ETags #63

I explain heroics (again) and then etags. If I didn’t know what etags are, I would have searched for them, likely read Wikipedia and look for some examples. You could ask for how to do what they’re asking (make requests to the Heroku API with etags) without Heroics as one possible path forwards.
I wanted to take the issue and expand on it. Instead, I decided to hold off and validate their request. I did mention my own needs/desires for a more consistent low-level interface.
I suggest closing the issue since “no one is actively working on it right now.”
- Instead of just closing, I provide an example of what could be useful to move the issue forwards: specific ideas of API design or example code, a PR, etc.
- Whenever I close an issue, I want to leave guidance for what might prompt me to re-open it (when relevant).
- Related to my real-life experience of “arguing effectively.” When I need to walk away from a situation/argument, it will feel rude and abrupt if you just leave. This could spark further conflict. Instead, I state that I’m leaving and making a plan for when we can revisit and come back to the issue.

Issue 10 UTF-8 with bom fails generating a client

Note: I’m purposefully not linking to this one because I got frustrated writing this response. I don’t want anyone posting anything negative to the thread. Later on, in the video, I enumerate some common types of seemingly “bad behavior” that might be attributed to other things such as English not being the first language (just an example, I don’t know this person at all). At the end of the day, my goal is to help the open-source project. I appreciate they took the time to open an issue. At this point in my marathon triaging session, I’m a little out of patience. I wanted to leave these issues in my videos as I think they represent real feelings and experiences you’ll have in the open-source community. Again: I don’t want to shame, or diss, or hate on this poster. Please take my comments in the general sense and with a grain of salt. Be nice. Be kind. Build community, not hate.

Update: I looked it up, and they’re from Saint Petersburg, so likely my read of “English not their first language” is likely correct.

I am frustrated by this issue. They were asked to provide more information and then didn’t describe the issue as “it is simple.”
- “99% of triaging issues is asking for reproductions/example-apps and verifying that they behave as described.”
- If you’re opening an issue, please be considerate of the maintainer’s time and energy. Don’t ask them to do things that you can do for them.
- If you’re frustrated with an interaction, try to center yourself on the common goals: In this case, everyone wants the open-source library to be better.
- If you must write a snarky response, don’t send it. Or maybe send it to a friend or something. Be nice. Issue reporters and commenters are contributing too. Empathy generates empathy, snark generates frustration and more snark.

While I’m somewhat accidentally on the topic of behavior that I don’t like: I encourage you to try to have empathy and understand if maybe there’s another way a comment could be interpreted. At the same time, if you see a pattern of bad behavior or a comment crosses a personal line, be protective of yourself and your community. 99% of the time that I’ve messaged someone privately saying, “when you said it made me feel , and that's bad," they apologized and fixed the behavior. This experience is where my comment of "maybe English is not their first language" came from. Most of those people I told them their words were hurtful had no idea and genuinely appreciated the opportunity to improve. For actual bad actors, there are mod tools. In general, I recommend the communication technique Non-Violent Communication - NVC. At a high level: state your observation, say how it made you feel, state what you require, explicitly state what you need from the person.

Stating, “I don’t know what this means” is difficult, especially on the internet. It’s helpful to state what you didn’t understand because if you have a question, it’s likely someone else does. By taking that risk, that leap of stating what wasn’t clear, you’re normalizing the community’s behavior.
Some people who create issues don’t know that not having a reproduction attached to an issue is the blocker to moving forwards. Sometimes you, as the issue triager, need to push back a little stronger. If they’ve already been dragging their feet, then stating, “let’s close this until we can get a reproduction” might motivate them to work on providing the needed example.

Issue 11 I dont have definitions in my json schemas #67

Note: Also not linking to this one due to my previous “frustration” comments. It’s the same person. Here’s where I first realize that English might not be their first language.

Triaging multiple issues in the same repo helps show you patterns and context you wouldn’t otherwise get. For example, the fact that one person created multiple issues. Even if they’re not explicitly linked via URLs, they might be related.
It’s okay to not understand an issue AT ALL when you read it. When this happens, you need to figure out if you want to invest the time and energy to learn more or cut your losses and move on to another issue. When you’re first getting started, you’ll be confused more often than you’ll know what you’re doing. One “hack” is to timebox your involvement with issues. I’m shooting for about 10 minutes here roughly. If, after that time, I still had no clue, I could choose to ask some clarifying questions possibly or to instead invest my energy in another issue.
When I said, “that’s what I would have said as well.” I’m referring to the statement by them to ask the poster to provide a PR for additional docs.
At this point, I mention the grammar of the issue and introduce the idea that maybe English is not their first language.
If you’re opening up an issue, try not to accidentally write in a commanding tone. Instead of “you should point to this problem in your readme,” maybe a softer, “the readme should point to this problem.” Describe the end goal instead of the action to get to the end goal.
Closing issues is better when you yell out “CLOSED” in a happy and accomplished voice.
If you didn’t know, you can edit the README or docs without ever needing to leave the web interface of GitHub.
Triaging issues is exhausting. My patience is wearing thin, and it’s showing in my responses. The more people who can help, the better responses and interactions the community will have.

At this point, there were 2 issues left. One of them was about deprecations, and I already wanted to work on it. I had previously commented on it. The last issue was one that we opened in a prior issue triaging session.

(Bonus) Paring pull-request session 5/4

If you’re going to deprecate a method, please provide a suitable replacement suggestion instead of “this method is obsolete.”
- URI.escape and URI.unescape are both deprecated in Ruby
- The docs recommend CGI.escape
I use Duck Duck Go as a search engine mostly because I don’t like Google AMP and want to use the same engine on my phone and desktop. With DDG, you can “fall back” to a google search by adding “g!” to your query, which sends you to google. I don’t use it often but need to here.
The decision to deprecate URI.escape is 10 years old.
Docs that come up from APIdoc often show up first but frequently have wrong info. I want ruby-doc.org instead when possible.
When I got to the CGI page on ruby-doc.org, I searched “escape” in the browser and couldn’t find that method. When that happens, sometimes the method comes from an included module, so I click through to the CGI::Util module. This module is where the method is defined.
I started to blindly replace URI.escape with CGI.escape in the code.
A tab I previously opened was linked to a PR that got rid of the warning. I used this to investigate how they fixed the issue. They used URI::DEFAULT_PARSER.escape.
When I ran into that constant, I wanted to investigate it because it was unfamiliar. I loaded up IRB to investigate.
I re-read their PR to see if they left more context or comments.
At this point, I thought since the docs recommended CGI, I was more comfortable using that.
I decided to test it in IRB since I already had it open.
CGI.escape is not a suitable replacement.

require 'cgi'
irb(main):003:0> CGI.escape("url here")
=> "url+here"
irb(main):004:0> URI.escape("url here")
(irb):4: warning: URI.escape is obsolete
=> "url%20here"

Note: that the results are different

I decided to use URI::DEFAULT_PARSER instead of CGI.
After making the change, I ran the tests to see if that got rid of all the deprecation warnings in the output.
I tried updating my gems to get rid of deprecation warnings. Investigating the turn I see it’s not had a new release since 2014, and the repo is marked as unmaintained.
Question: Are we using this library that is marked as unmaintained? Removing the gem gives us an error.
At this point, I don’t know what it does and don’t know how long it will take to remove it. I decided to clean up the other warnings and save that for last if there’s time.
For the warning “assigned but unused,” you can use the character underscore _ as a variable to tell Ruby that you don’t care about using the variable.
For the duplicate test method, my initial thought is someone copied and pasted the whole method again by accident. On closer investigation, they’re similar but not the same. I rename one of the methods.
You can start a commit message with [close #<issue number>], and when it’s merged into the default branch, it auto closes that issue. It’s also helpful for bookkeeping later to ensure it’s clear the issue and the PR are related.
When writing a commit message, try to give the maximum amount of context. Imagine if you had amnesia and found yourself in front of the commit message and need to understand why you did what you did. In this case, I wanted to link to external resources. I also realized I didn’t read through the history of why URI.escape is deprecated and might want to later, so I use the commit as a place to store the link.
Usually, when I make a PR, I bundle all the changes into one commit via squashing my commits. Conventions vary based on the project. Some people try to “pad” their contribution with extra commits, so it looks like they’re more active/helpful than they really are on a repo. Don’t do that. Do what’s right for the conventions of the community you’re working with and what’s best for the project. I said “rebase” but meant “squash.”
Since I don’t know how deeply integrated turn is into the project, I want to timebox removing it.
I deleted the references to turn, and the tests pass.
It might seem redundant to re-write a good PR message since there are already links to the commits with individual contexts. It’s worth spending the extra time and writing good PR messages and descriptions, so all the appropriate context is available with minimal maintainer effort.
Notice that I’m re-using my notes from working on the issue while making the PR.
It’s not guaranteed that triaging issues will show you pull request opportunities, but it’s an excellent place to start looking and start building context.

Wrap Up

If you’re inspired to go work on some open-source issues and don’t know where to get started, I recommend signing up for CodeTriage. If you like this “with me” series, find me on twitter @schneems and pitch me what you would like for me to work on live and record for another session. You can also subscribe to my email newsletter to get more content.

The Life-Changing Magic of Tidying Ruby Object Allocations

Wed, 16 Sep 2020 00:00:00 +0000

Your app is slow. It does not spark joy. This post will show you how to use memory allocation profiling tools to discover performance hotspots, even when they’re coming from inside a library. We will use this technique with a real-world application to identify a piece of optimizable code in Active Record that ultimately leads to a patch with a substantial impact on page speed.

Read the original post on the Heroku engineering blog

In addition to the talk, I’ve gone back and written a full technical recap of each section to revisit it any time you want without going through the video.

I make heavy use of theatrics here, including a Japanese voiceover artist, animoji, and some edited clips of Marie Kondo’s Netflix TV show. This recording was done at EuRuKo on a boat. If you’ve got the time, here’s the talk:

Intro to Tidying Object Allocations

The core premise of this talk is that we all want faster applications. Here I’m making the pitch that you can get significant speedups by focusing on your object allocations. To do that, I’ll eventually show you a few real-world cases of PRs I made to Rails along with a “how-to” that shows how I used profiling and benchmarking to find and fix the hotspots.

At a high level, the “tidying” technique looks like this:

Take all your object allocations and put them in a pile where you can see them
Consider each one: Does it spark joy?
Keep only the objects that spark joy

An object sparks joy if it is useful, keeps your code clean, and does not cause performance problems. If an object is absolutely necessary, and removing it causes your code to crash, it sparks joy.

To put object allocations in front of us we’ll use:

To get a sense of the cost of object allocation, we can benchmark two different ways to perform the same logic. One of these allocates an array while the other does not.

require 'benchmark/ips'

def compare_max(a, b)
  return a if a > b
  b
end

def allocate_max(a, b)
  array = [a, b] # <===== Array allocation here
  array.max
end

Benchmark.ips do |x|
  x.report("allocate_max") {
    allocate_max(1, 2)
  }
  x.report("compare_max ") {
    compare_max(1, 2)
  }
  x.compare!
end

This gives us the results:

Warming up --------------------------------------
        allocate_max   258.788k i/100ms
        compare_max    307.196k i/100ms
Calculating -------------------------------------
        allocate_max      6.665M (±14.6%) i/s -     32.090M in   5.033786s
        compare_max      13.597M (± 6.0%) i/s -     67.890M in   5.011819s

Comparison:
        compare_max : 13596747.2 i/s
        allocate_max:  6664605.5 i/s - 2.04x  slower

In this example, allocating an array is 2x slower than making a direct comparison. It’s a truism in most languages that allocating memory or creating objects is slow. In the C programming language, it’s a truism that “malloc is slow.”

Since we know that allocating in Ruby is slow, we can make our programs faster by removing allocations. As a simplifying assumption, I’ve found that a decrease in bytes allocated roughly corresponds to performance improvement. For example, if I can reduce the number of bytes allocated by 1% in a request, then on average, the request will have been sped up by about 1%. This assumption helps us benchmark faster as it’s much easier to measure memory allocated than it is to repeatedly run hundreds or thousands of timing benchmarks.

Tidying Example 1: Active Record `respond_to?` logic

Using the target application CodeTriage.com and derailed benchmarks, we get a “pile” of memory allocations:

$ bundle exec derailed exec perf:objects

allocated memory by gem
-----------------------------------
    227058  activesupport/lib
    134366  codetriage/app
    # ...


allocated memory by file
-----------------------------------
    126489  …/code/rails/activesupport/lib/active_support/core_ext/string/output_safety.rb
     49448  …/code/codetriage/app/views/layouts/_app.html.slim
     49328  …/code/codetriage/app/views/layouts/application.html.slim
     36097  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb
     25096  …/code/codetriage/app/views/pages/_repos_with_pagination.html.slim
     24432  …/code/rails/activesupport/lib/active_support/core_ext/object/to_query.rb
     23526  …/code/codetriage/.gem/ruby/2.5.3/gems/rack-mini-profiler-1.0.0/lib/patches/db/pg.rb
     21912  …/code/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
     18000  …/code/rails/activemodel/lib/active_model/attribute_set/builder.rb
     15888  …/code/rails/activerecord/lib/active_record/result.rb
     14610  …/code/rails/activesupport/lib/active_support/cache.rb
     11109  …/code/codetriage/.gem/ruby/2.5.3/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/storage/file_store.rb
      9824  …/code/rails/actionpack/lib/abstract_controller/caching/fragments.rb
      9360  …/.rubies/ruby-2.5.3/lib/ruby/2.5.0/logger.rb
      8440  …/code/rails/activerecord/lib/active_record/attribute_methods.rb
      8304  …/code/rails/activemodel/lib/active_model/attribute.rb
      8160  …/code/rails/actionview/lib/action_view/renderer/partial_renderer.rb
      8000  …/code/rails/activerecord/lib/active_record/integration.rb
      7880  …/code/rails/actionview/lib/action_view/log_subscriber.rb
      7478  …/code/rails/actionview/lib/action_view/helpers/tag_helper.rb
      7096  …/code/rails/actionview/lib/action_view/renderer/partial_renderer/collection_caching.rb
      # ...

The full output is massive, so I’ve truncated it here.

Once you’ve got your memory in a pile. I like to look at the “allocated memory” by file. I start at the top and look at each in turn. In this case, we’ll look at this file:

      8440  …/code/rails/activerecord/lib/active_record/attribute_methods.rb

Once you have a file you want to look at, you can focus on it in derailed like this:

$ ALLOW_FILES=active_record/attribute_methods.rb \
  bundle exec derailed exec perf:objects

allocated memory by file
-----------------------------------
      8440  …/code/rails/activerecord/lib/active_record/attribute_methods.rb

allocated memory by location
-----------------------------------
      8000  …/code/rails/activerecord/lib/active_record/attribute_methods.rb:270
       320  …/code/rails/activerecord/lib/active_record/attribute_methods.rb:221
        80  …/code/rails/activerecord/lib/active_record/attribute_methods.rb:189
        40  …/code/rails/activerecord/lib/active_record/attribute_methods.rb:187

Now we can see exactly where the memory is being allocated in this file. Starting at the top of the locations, I’ll work my way down to understand how memory is allocated and used. Looking first at this line:

      8000  …/code/rails/activerecord/lib/active_record/attribute_methods.rb:270

We can open this in an editor and navigate to that location:

$ bundle open activerecord

In that file, here’s the line allocating the most memory:

def respond_to?(name, include_private = false)
  return false unless super

  case name
  when :to_partial_path
    name = "to_partial_path"
  when :to_model
    name = "to_model"
  else
    name = name.to_s # <=== Line 270 here
  end

  # If the result is true then check for the select case.
  # For queries selecting a subset of columns, return false for unselected columns.
  # We check defined?(@attributes) not to issue warnings if called on objects that
  # have been allocated but not yet initialized.
  if defined?(@attributes) && self.class.column_names.include?(name)
    return has_attribute?(name)
  end

  true
end

Here we can see on line 270 that it’s allocating a string. But why? To answer that question, we need more context. We need to understand how this code is used. When we call respond_to on an object, we want to know if a method by that name exists. Because Active Record is backed by a database, it needs to see if a column exists with that name.

Typically when you call respond_to you pass in a symbol, for example, user.respond_to?(:email). But in Active Record, columns are stored as strings. On line 270, we’re ensuring that the name value is always a string.

This is the code where name is used:

  if defined?(@attributes) && self.class.column_names.include?(name)

Here column_names returns an array of column names, and the include? method will iterate over each until it finds the column with that name, or its nothing (nil).

To determine if we can get rid of this allocation, we have to figure out if there’s a way to replace it without allocating memory. We need to refactor this code while maintaining correctness. I decided to add a method that converted the array of column names into a hash with symbol keys and string values:

# lib/activerecord/model_schema.rb
def symbol_column_to_string(name_symbol) # :nodoc:
  @symbol_column_to_string_name_hash ||= column_names.index_by(&:to_sym)
  @symbol_column_to_string_name_hash[name_symbol]
end

This is how you would use it:

User.symbol_column_to_string(:email) #=> "email"
User.symbol_column_to_string(:foo)   #=> nil

Since the value that is being returned every time by this method is from the same hash, we can re-use the same string and not have to allocate. The refactored respond_to code ends up looking like this:

def respond_to?(name, include_private = false)
  return false unless super

  # If the result is true then check for the select case.
  # For queries selecting a subset of columns, return false for unselected columns.
  # We check defined?(@attributes) not to issue warnings if called on objects that
  # have been allocated but not yet initialized.
  if defined?(@attributes)
    if name = self.class.symbol_column_to_string(name.to_sym)
      return has_attribute?(name)
    end
  end

  true
end

Running our benchmarks, this patch yielded a reduction in memory of 1%. Using code that eventually became derailed exec perf:library, I verified that the patch made end-to-end request/response page speed on CodeTriage 1% faster.

Performance and Statistical Significance

When talking about benchmarks, it’s important to talk about statistics and their impact. I talk a bit about this in Lies, Damned Lies, and Averages: Perc50, Perc95 explained for Programmers. Essentially any time you measure a value, there’s a chance that it could result from randomness. If you run a benchmark 3 times, it will give you 3 different results. If it shows that it was faster twice and slower once, how can you be certain that the results are because of the change and not random chance?

That’s precisely the question that “statistical significance” tries to answer. While we can never know, we can make an informed decision. How? Well, if you took a measurement of the same code many times, you would know any variation was the result of randomness. This would give you a distribution of randomness. Then you could use this distribution to understand how likely it is that your change was caused by randomness.

In the talk, I go into detail on the origins of “Student’s T-Test.” In derailed, I’ve switched to using Kolmogorov-Smirnov instead. When I ran benchmarks on CodeTriage, I wanted to be sure that my results were valid, so I ran them multiple times and ran Kolmogorov Smirnov on them. This gives me a confidence interval. If my results are in that interval, then I can say with 95% certainty that my results are not the result of random chance i.e., that they’re valid and are statistically significant.

If it’s not significant, it could mean that the change is too small to detect, that you need more samples, or that there is no difference.

In addition to running a significance check on your change, it’s useful to see the distribution. Derailed benchmarks does this for you by default now. Here is a result from derailed exec perf:library used to compare the performance difference of two different commits in a library dependency:

                  Histogram - [winner] "I am the new commit."
                           ┌                                        ┐
            [11.2 , 11.28) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 12
            [11.28, 11.36) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 22
            [11.35, 11.43) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 30
            [11.43, 11.51) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 17
   Time (s) [11.5 , 11.58) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 13
            [11.58, 11.66) ┤▇▇▇▇▇▇▇ 6
            [11.65, 11.73) ┤ 0
            [11.73, 11.81) ┤ 0
            [11.8 , 11.88) ┤ 0
                           └                                        ┘
                                      # of runs in range



                  Histogram - [loser] "Old commit"
                           ┌                                        ┐
            [11.2 , 11.28) ┤▇▇▇▇ 3
            [11.28, 11.36) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 19
            [11.35, 11.43) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 17
            [11.43, 11.51) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 25
   Time (s) [11.5 , 11.58) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 15
            [11.58, 11.66) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 13
            [11.65, 11.73) ┤▇▇▇▇ 3
            [11.73, 11.81) ┤▇▇▇▇ 3
            [11.8 , 11.88) ┤▇▇▇ 2
                           └                                        ┘
                                      # of runs in range

The TLDR of this whole section is that in addition to showing my change as being faster, I was also able to show that the improvement was statistically significant.

Tidying example 2: Converting strings to time takes time

One percent faster is good, but it could be better. Let’s do it again. First, get a pile of objects:

$ bundle exec derailed exec perf:objects

# ...

allocated memory by file
-----------------------------------
    126489  …/code/rails/activesupport/lib/active_support/core_ext/string/output_safety.rb
     49448  …/code/codetriage/app/views/layouts/_app.html.slim
     49328  …/code/codetriage/app/views/layouts/application.html.slim
     36097  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb
     25096  …/code/codetriage/app/views/pages/_repos_with_pagination.html.slim
     24432  …/code/rails/activesupport/lib/active_support/core_ext/object/to_query.rb
     23526  …/code/codetriage/.gem/ruby/2.5.3/gems/rack-mini-profiler-1.0.0/lib/patches/db/pg.rb
     21912  …/code/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
     18000  …/code/rails/activemodel/lib/active_model/attribute_set/builder.rb
     15888  …/code/rails/activerecord/lib/active_record/result.rb
     14610  …/code/rails/activesupport/lib/active_support/cache.rb
     11148  …/code/codetriage/.gem/ruby/2.5.3/gems/rack-mini-profiler-1.0.0/lib/mini_profiler/storage/file_store.rb
      9824  …/code/rails/actionpack/lib/abstract_controller/caching/fragments.rb
      9360  …/.rubies/ruby-2.5.3/lib/ruby/2.5.0/logger.rb
      8304  …/code/rails/activemodel/lib/active_model/attribute.rb

Zoom in on a file:

     36097  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb

Isolate the file:

$ ALLOW_FILE=active_model/type/helpers/time_value.rb \
  bundle exec derailed exec perf:objects

Total allocated: 39617 bytes (600 objects)
Total retained:  0 bytes (0 objects)

allocated memory by gem
-----------------------------------
     39617  activemodel/lib

allocated memory by file
-----------------------------------
     39617  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb

allocated memory by location
-----------------------------------
     17317  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb:72
     12000  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb:74
      6000  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb:73
      4300  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb:64

We’re going to do the same thing by starting to look at the top location:

     17317  …/code/rails/activemodel/lib/active_model/type/helpers/time_value.rb:72

Here’s the code:

def fast_string_to_time(string)
 if string =~ ISO_DATETIME # <=== line 72 Here
   microsec = ($7.to_r * 1_000_000).to_i
   new_time $1.to_i, $2.to_i, $3.to_i, $4.to_i, $5.to_i, $6.to_i, microsec
 end
end

On line 72, we are matching the input string with a regular expression constant. This allocates a lot of memory because each grouped match of the regular expression allocates a new string. To understand if we can make this faster, we have to understand how it’s used.

This method takes in a string, then uses a regex to split it into parts, and then sends those parts to the new_time method.

There’s not much going on that can be sped up there, but what’s happening on this line:

   microsec = ($7.to_r * 1_000_000).to_i

Here’s the regex:

ISO_DATETIME = /\A(\d{4})-(\d\d)-(\d\d) (\d\d):(\d\d):(\d\d)(\.\d+)?\z/

When I ran the code and output $7 from the regex match, I found that it would contain a string that starts with a dot and then has numbers, for example:

puts $7 # => ".1234567"

This code wants microseconds as an integer, so it turns it into a “rational” and then multiplies it by a million and turns it into an integer.

($7.to_r * 1_000_000).to_i # => 1234567

You might notice that it looks like we’re basically dropping the period and then turning it into an integer. So why not do that directly?

Here’s what it looks like:

def fast_string_to_time(string)
  if string =~ ISO_DATETIME
    microsec_part = $7
    if microsec_part && microsec_part.start_with?(".") && microsec_part.length == 7
      microsec_part[0] = ""         # <=== HERE
      microsec = microsec_part.to_i # <=== HERE
    else
      microsec = (microsec_part.to_r * 1_000_000).to_i
    end
    new_time $1.to_i, $2.to_i, $3.to_i, $4.to_i, $5.to_i, $6.to_i, microsec
  end

We’ve got to guard this case by checking for the conditions of our optimization. Now the question is: is this faster?

Here’s a microbenchmark:

original_string = ".443959"

require 'benchmark/ips'

Benchmark.ips do |x|
  x.report("multiply") {
    string = original_string.dup
    (string.to_r * 1_000_000).to_i
  }
  x.report("new     ") {
    string = original_string.dup
    if string && string.start_with?(".".freeze) && string.length == 7
      string[0] = ''.freeze
      string.to_i
    end
  }
  x.compare!
end

# Warming up --------------------------------------
#             multiply   125.783k i/100ms
#             new        146.543k i/100ms
# Calculating -------------------------------------
#             multiply      1.751M (± 3.3%) i/s -      8.805M in   5.033779s
#             new           2.225M (± 2.1%) i/s -     11.137M in   5.007110s

# Comparison:
#             new     :  2225289.7 i/s
#             multiply:  1751254.2 i/s - 1.27x  slower

The original code is 1.27x slower. YAY!

Tidying Example 3: Lightning fast cache keys

The last speedup is kind of underwhelming, so you might wonder why I added it. If you remember our first example of optimizing respond_to, it helped to understand the broader context of how it’s used. Since this is such an expensive object allocation location, is there an opportunity to call it less or not call it at all?

To find out, I added a puts caller in the code and re-ran it. Here’s part of a backtrace:

====================================================================================================
…/code/rails/activemodel/lib/active_model/type/date_time.rb:25:in `cast_value'
…/code/rails/activerecord/lib/active_record/connection_adapters/postgresql/oid/date_time.rb:16:in `cast_value'
…/code/rails/activemodel/lib/active_model/type/value.rb:38:in `cast'
…/code/rails/activemodel/lib/active_model/type/helpers/accepts_multiparameter_time.rb:12:in `block in initialize'
…/code/rails/activemodel/lib/active_model/type/value.rb:24:in `deserialize'
…/.rubies/ruby-2.5.3/lib/ruby/2.5.0/delegate.rb:349:in `block in delegating_block'
…/code/rails/activerecord/lib/active_record/attribute_methods/time_zone_conversion.rb:8:in `deserialize'
…/code/rails/activemodel/lib/active_model/attribute.rb:164:in `type_cast'
…/code/rails/activemodel/lib/active_model/attribute.rb:42:in `value'
…/code/rails/activemodel/lib/active_model/attribute_set.rb:48:in `fetch_value'
…/code/rails/activerecord/lib/active_record/attribute_methods/read.rb:77:in `_read_attribute'
…/code/rails/activerecord/lib/active_record/attribute_methods/read.rb:40:in `__temp__57074616475646f51647'
…/code/rails/activesupport/lib/active_support/core_ext/object/try.rb:16:in `public_send'
…/code/rails/activesupport/lib/active_support/core_ext/object/try.rb:16:in `try'
…/code/rails/activerecord/lib/active_record/integration.rb:99:in `cache_version'
…/code/rails/activerecord/lib/active_record/integration.rb:68:in `cache_key'
…/code/rails/activesupport/lib/active_support/cache.rb:639:in `expanded_key'
…/code/rails/activesupport/lib/active_support/cache.rb:644:in `block in expanded_key'
…/code/rails/activesupport/lib/active_support/cache.rb:644:in `collect'
…/code/rails/activesupport/lib/active_support/cache.rb:644:in `expanded_key'
…/code/rails/activesupport/lib/active_support/cache.rb:608:in `normalize_key'
…/code/rails/activesupport/lib/active_support/cache.rb:565:in `block in read_multi_entries'
…/code/rails/activesupport/lib/active_support/cache.rb:564:in `each'
…/code/rails/activesupport/lib/active_support/cache.rb:564:in `read_multi_entries'
…/code/rails/activesupport/lib/active_support/cache.rb:387:in `block in read_multi'

I followed it backwards until I hit these two places:

…/code/rails/activerecord/lib/active_record/integration.rb:99:in `cache_version'
…/code/rails/activerecord/lib/active_record/integration.rb:68:in `cache_key'

It looks like this expensive code is being called while generating a cache key.

def cache_key(*timestamp_names)
  if new_record?
    "#{model_name.cache_key}/new"
  else
    if cache_version && timestamp_names.none? # <== line 68 here
      "#{model_name.cache_key}/#{id}"
    else
      timestamp = if timestamp_names.any?
        ActiveSupport::Deprecation.warn(<<-MSG.squish)
          Specifying a timestamp name for #cache_key has been deprecated in favor of
          the explicit #cache_version method that can be overwritten.
        MSG

        max_updated_column_timestamp(timestamp_names)
      else
        max_updated_column_timestamp
      end

      if timestamp
        timestamp = timestamp.utc.to_s(cache_timestamp_format)
        "#{model_name.cache_key}/#{id}-#{timestamp}"
      else
        "#{model_name.cache_key}/#{id}"
      end
    end
  end
end

On line 68 in the cache_key code it calls cache_version. Here’s the code for cache_version:

def cache_version # <== line 99 here
  if cache_versioning && timestamp = try(:updated_at)
    timestamp.utc.to_s(:usec)
  end
end

Here is our culprit:

timestamp = try(:updated_at)

What is happening is that some database adapters, such as the one for Postgres, returned their values from the database driver as strings. Then active record will lazily cast them into Ruby objects when they are needed. In this case, our time value method is being called to convert the updated timestamp into a time object so we can use it to generate a cache version string.

Here’s the value before it’s converted:

User.first.updated_at_before_type_cast # => "2019-04-24 21:21:09.232249"

And here’s the value after it’s converted:

User.first.updated_at.to_s(:usec)      # => "20190424212109232249"

Basically, all the code is doing is trimming out the non-integer characters. Like before, we need a guard that our optimization can be applied:

# Detects if the value before type cast
# can be used to generate a cache_version.
#
# The fast cache version only works with a
# string value directly from the database.
#
# We also must check if the timestamp format has been changed
# or if the timezone is not set to UTC then
# we cannot apply our transformations correctly.
def can_use_fast_cache_version?(timestamp)
  timestamp.is_a?(String) &&
    cache_timestamp_format == :usec &&
    default_timezone == :utc &&
    !updated_at_came_from_user?
end

Then once we’re in that state, we can modify the string directly:

# Converts a raw database string to `:usec`
# format.
#
# Example:
#
#   timestamp = "2018-10-15 20:02:15.266505"
#   raw_timestamp_to_cache_version(timestamp)
#   # => "20181015200215266505"
#
# PostgreSQL truncates trailing zeros,
# https://github.com/postgres/postgres/commit/3e1beda2cde3495f41290e1ece5d544525810214
# to account for this we pad the output with zeros
def raw_timestamp_to_cache_version(timestamp)
  key = timestamp.delete("- :.")
  if key.length < 20
    key.ljust(20, "0")
  else
    key
  end
end

There’s some extra logic due to the Postgres truncation behavior linked above. The resulting code to cache_version becomes:

def cache_version
  return unless cache_versioning

  if has_attribute?("updated_at")
    timestamp = updated_at_before_type_cast
    if can_use_fast_cache_version?(timestamp)
      raw_timestamp_to_cache_version(timestamp)
    elsif timestamp = updated_at
      timestamp.utc.to_s(cache_timestamp_format)
    end
  end
end

That’s the opportunity. What’s the impact?

Before: Total allocated: 743842 bytes (6626 objects)
After:  Total allocated: 702955 bytes (6063 objects)

The bytes reduced is 5% fewer allocations. Which is pretty good. How does it translate to speed?

It turns out that time conversion is very CPU intensive and changing this code makes the target application up to 1.12x faster. This means that if your app previously required 100 servers to run, it can now run with about 88 servers.

Wrap up

Adding together these optimizations and others brings the overall performance improvement to 1.23x or a net reduction of 19 servers. Basically, it’s like buying 4 servers and getting 1 for free.

These examples were picked from my changes to the Rails codebase, but you can use them to optimize your applications. The general framework looks like this:

Get a list of all your memory
Zoom in on a hotspot
Figure out what is causing that memory to be allocated inside of your code
Ask if you can refactor your code to not depend on those allocations

If you want to learn more about memory, here are my recommendations:

Why does my App’s Memory Use Grow Over Time? - Start here, an excellent high-level overview of what causes a system’s memory to grow that will help you develop an understanding of how Ruby allocates and uses memory at the application level.
Complete Guide to Rails Performance (Book) - This book is by Nate Berkopec and is excellent. I recommend it to someone at least once a week.
How Ruby uses memory - This is a lower level look at precisely what “retained” and “allocated” memory means. It uses small scripts to demonstrate Ruby memory behavior. It also explains why the “total max” memory of our system rarely goes down.
How Ruby uses memory (Video) - If you’re new to the concepts of object allocation, this might be an excellent place to start (you can skip the first story in the video, the rest are about memory). Memory stuff starts at 13 minutes
Jumping off the Ruby Memory Cliff - Sometimes you might see a ‘cliff’ in your memory metrics or a saw-tooth pattern. This article explores why that behavior exists and what it means.
Ruby Memory Use (Heroku Devcenter article I maintain) - This article focuses on alleviating the symptoms of high memory use.
Debugging a memory leak on Heroku - TLDR; It’s probably not a leak. Still worth reading to see how you can come to the same conclusions yourself. Content is valid for other environments than Heroku. Lots of examples of using the tool derailed_benchmarks (that I wrote).
The Life-Changing Magic of Tidying Active Record Allocations (Post + Video) - This talk shows how I used tools to track down and eliminate memory allocations in real life. All of the examples are from patches I submitted to Rails, but the process works the same for finding allocations caused by your application logic.

Get ahold of Richard and stay up-to-date with Ruby, Rails, and other programming related content through a subscription to his mailing list.

A Fast Car Needs Good Brakes: How We Added Client Rate Throttling to the Platform API Gem

Wed, 08 Jul 2020 00:00:00 +0000

When API requests are made one-after-the-other they’ll quickly hit rate limits and when that happens:

If you provide an API client that doesn't include rate limiting, you don't really have an API client. You've got an exception generator with a remote timer.
— Richard Schneeman 🤠 Stay Inside (@schneems) June 12, 2019

That tweet spawned a discussion that generated a quest to add rate throttling logic to the platform-api gem that Heroku maintains for talking to its API in Ruby.

This blog post is originally published on the Heroku Engineering Blog

If the term “rate throttling” is new to you, read Rate limiting, rate throttling, and how they work together

The Heroku API uses Genetic Cell Rate Algorithm (GCRA) as described by Brandur in this post on the server-side. Heroku’s API docs state:

The API limits the number of requests each user can make per hour to protect against abuse and buggy code. Each account has a pool of request tokens that can hold at most 4500 tokens. Each API call removes one token from the pool. Tokens are added to the account pool at a rate of roughly 75 per minute (or 4500 per hour), up to a maximum of 4500. If no tokens remain, further calls will return 429 Too Many Requests until more tokens become available.

I needed to write an algorithm that never errored as a result of a 429 response. A “simple” solution would be to add a retry to all requests when they see a 429, but that would effectively DDoS the API. I made it a goal for the rate throttling client also to minimize its retry rate. That is, if the client makes 100 requests, and 10 of them are a 429 response that its retry rate is 10%. Since the code needed to be contained entirely in the client library, it needed to be able to function without distributed coordination between multiple clients on multiple machines except for whatever information the Heroku API returned.

Making client throttling maintainable

Before we can get into what logic goes into a quality rate throttling algorithm, I want to talk about the process that I used as I think the journey is just as fascinating as the destination.

I initially started wanting to write tests for my rate throttling strategy. I quickly realized that while testing the behavior “retries a request after a 429 response,” it is easy to check. I also found that checking for quality “this rate throttle strategy is better than others” could not be checked quite as easily. The solution that I came up with was to write a simulator in addition to tests. I would simulate the server’s behavior, and then boot up several processes and threads and hit the simulated server with requests to observe the system’s behavior.

I initially just output values to the CLI as the simulation ran, but found it challenging to make sense of them all, so I added charting. I found my simulation took too long to run and so I added a mechanism to speed up the simulated time. I used those two outputs to write what I thought was a pretty good rate throttling algorithm. The next task was wiring it up to the platform-api gem.

To help out I paired with a Heroku Engineer, Lola, we ended up making several PRs to a bunch of related projects, and that’s its own story to tell. Finally, the day came where we were ready to get rate throttling into the platform-api gem; all we needed was a review.

Unfortunately, the algorithm I developed from “watching some charts for a few hours” didn’t make a whole lot of sense, and it was painfully apparent that it wasn’t maintainable. While I had developed a good gut feel for what a “good” algorithm did and how it behaved, I had no way of solidifying that knowledge into something that others could run with. Imagine someone in the future wants to make a change to the algorithm, and I’m no longer here. The tests I had could prevent them from breaking some expectations, but there was nothing to help them make a better algorithm.

The making of an algorithm

At this point, I could explain the approach I had taken to build an algorithm, but I had no way to quantify the “goodness” of my algorithm. That’s when I decided to throw it all away and start from first principles. Instead of asking “what would make my algorithm better,” I asked, “how would I know a change to my algorithm is better” and then worked to develop some ways to quantify what “better” meant. Here are the goals I ended up coming up with:

Minimize average retry rate: The fewer failed API requests, the better
Minimize maximum sleep time: Rate throttling involves waiting, and no one wants to wait for too long
Minimize variance of request count between clients: No one likes working with a greedy co-worker, API clients are no different. No client in the distributed system should be an extended outlier
Minimize time to clear a large request capacity: As the system changes, clients should respond quickly to changes.

I figured that if I could generate metrics on my rate-throttle algorithm and compare it to simpler algorithms, then I could show why individual decisions were made.

I moved my hacky scripts for my simulation into a separate repo and, rather than relying on watching charts and logs, moved to have my simulation produce numbers that could be used to quantify and compare algorithms.

With that work under my belt, I threw away everything I knew about rate-throttling and decided to use science and measurement to guide my way.

Writing a better rate-throttling algorithm with science: exponential backoff

Earlier I mentioned that a “simple” algorithm would be to retry requests. A step up in complexity and functionality would be to retry requests after an exponential backoff. I coded it up and got some numbers for a simulated 30-minute run (which takes 3 minutes of real-time):

Avg retry rate:      60.08 %
Max sleep time:      854.89 seconds
Stdev Request Count: 387.82

Time to clear workload (4500 requests, starting_sleep: 1s):
74.23 seconds

Now that we’ve got baseline numbers, how could we work to minimize any of these values? In my initial exponential backoff model, I multiplied sleep by a factor of 2.0, what would happen if I increased it to 3.0 or decreased it to 1.2?

To find out, I plugged in those values and re-ran my simulations. I found that there was a correlation between retry rate and max sleep value with the backoff factor, but they were inverse. I could lower the retry rate by increasing the factor (to 3.0), but this increased my maximum sleep time. I could reduce the maximum sleep time by decreasing the factor (to 1.2), but it increased my retry rate.

That experiment told me that if I wanted to optimize both retry rate and sleep time, I could not do it via only changing the exponential factor since an improvement in one meant a degradation in the other value.

At this point, we could theoretically do anything, but our metrics judge our success. We could put a cap on the maximum sleep time, for example, we could write code that says “don’t sleep longer than 300 seconds”, but it too would hurt the retry rate. The biggest concern for me in this example is the maximum sleep time, 854 seconds is over 14 minutes which is WAAAYY too long for a single client to be sleeping.

I ended up picking the 1.2 factor to decrease that value at the cost of a worse retry-rate:

Avg retry rate:      80.41 %
Max sleep time:      46.72 seconds
Stdev Request Count: 147.84

Time to clear workload (4500 requests, starting_sleep: 1s):
74.33 seconds

Forty-six seconds is better than 14 minutes of sleep by a long shot. How could we get the retry rate down?

Incremental improvement: exponential sleep with a gradual decrease

In the exponential backoff model, it backs-off once it sees a 429, but as soon as it hits a success response, it doesn’t sleep at all. One way to reduce the retry-rate would be to assume that once a request had been rate-throttled, that future requests would need to wait as well. Essentially we would make the sleep value “sticky” and sleep before all requests. If we only remembered the sleep value, our rate throttle strategy wouldn’t be responsive to any changes in the system, and it would have a poor “time to clear workload.” Instead of only remembering the sleep value, we can gradually reduce it after every successful request. This logic is very similar to TCP slow start.

How does it play out in the numbers?

Avg retry rate:      40.56 %
Max sleep time:      139.91 seconds
Stdev Request Count: 867.73

Time to clear workload (4500 requests, starting_sleep: 1s):
115.54 seconds

Retry rate did go down by about half. Sleep time went up, but it’s still well under the 14-minute mark we saw earlier. But there’s a problem with a metric I’ve not talked about before, the “stdev request count.” It’s easier to understand if you look at a chart to see what’s going on:

Here you can see one client is sleeping a lot (the red client) while other clients are not sleeping at all and chewing through all the available requests at the bottom. Not all the clients are behaving equitably. This behavior makes it harder to tune the system.

One reason for this inequity is that all clients are decreasing by the same constant value for every successful request. For example, let’s say we have a client A that is sleeping for 44 seconds, and client B that is sleeping for 11 seconds and both decrease their sleep value by 1 second after every request. If both clients ran for 45 seconds, it would look like this:

Client A) Sleep 44 (Decrease value: 1)
Client B) Sleep 11 (Decrease value: 1)
Client B) Sleep 10 (Decrease value: 1)
Client B) Sleep  9 (Decrease value: 1)
Client B) Sleep  8 (Decrease value: 1)
Client B) Sleep  7 (Decrease value: 1)
Client A) Sleep 43 (Decrease value: 1)

So while client A has decreased by 1 second total, client B has reduced by 4 seconds total, since it is firing 4x as fast (i.e., it’s sleep time is 4x lower). So while the decrease rate is equal, it is not equitable. Ideally, we would want all clients to decrease at the same rate.

All clients created equal: exponential increase proportional decrease

Since clients cannot communicate with each other in our distributed system, one way to guaranteed proportional decreases is to use the sleep value in the decrease amount:

decrease_value = (sleep_time) / some_value

Where some_value is a magic number. In this scenario the same clients A and B running for 45 seconds would look like this with a value of 100:

Client A) Sleep 44
Client B) Sleep 11
Client B) Sleep 10.89 (Decrease value: 11.00/100 = 0.1100)
Client B) Sleep 10.78 (Decrease value: 10.89/100 = 0.1089)
Client B) Sleep 10.67 (Decrease value: 10.78/100 = 0.1078)
Client B) Sleep 10.56 (Decrease value: 10.67/100 = 0.1067)
Client A) Sleep 43.56 (Decrease value: 44.00/100 = 0.4400)

Now client A has had a decrease of 0.44, and client B has had a reduction of 0.4334 (11 seconds - 10.56 seconds), which is a lot more equitable than before. Since some_value is tunable, I wanted to use a larger number so that the retry rate would be lower than 40%. I chose 4500 since that’s the maximum number of requests in the GCRA bucket for Heroku’s API.

Here’s what the results looked like:

Avg retry rate:      3.66 %
Max sleep time:      17.31 seconds
Stdev Request Count: 101.94

Time to clear workload (4500 requests, starting_sleep: 1s):
551.10 seconds

The retry rate went WAAAY down, which makes sense since we’re decreasing slower than before (the constant decrease value previously was 0.8). Stdev went way down as well. It’s about 8x lower. Surprisingly the max sleep time went down as well. I believe this to be a factor of a decrease in the number of required exponential backoff events. Here’s what this algorithm looks like:

The only problem here is that the “time to clear workload” is 5x higher than before. What exactly is being measured here? In this scenario, we’re simulating a cyclical workflow where clients are running under high load, then go through a light load, and then back to a high load. The simulation starts all clients with a sleep value, but the server’s rate-limit is reset to 4500. The time is how long it takes the client to clear all 4500 requests.

What this metric of 551 seconds is telling me is that this strategy is not very responsive to a change in the system. To illustrate this problem, I ran the same algorithm starting each client at 8 seconds of sleep instead of 1 second to see how long it would take to trigger a rate limit:

The graph shows that it takes about 7 hours to clear all these requests, which is not good. What we need is a way to clear requests faster when there are more requests.

The only remaining option: exponential increase proportional remaining decrease

When you make a request to the Heroku API, it tells you how many requests you have left remaining in your bucket in a header. Our problem with the “proportional decrease” is mostly that when there are lots of requests remaining in the bucket, it takes a long time to clear them (if the prior sleep rate was high, such as in a varying workload). To account for this, we can decrease the sleep value quicker when the remaining bucket is full and slower when the remaining bucket is almost empty. To express that in an expression, it might look like this:

decrease_value = (sleep_time * request_count_remaining) / some_value

In my case, I chose some_value to be the maximum number of requests possible in a bucket, which is 4500. You can imagine a scenario where workers were very busy for a period and being rate limited. Then no jobs came in for over an hour - perhaps the workday was over, and the number of requests remaining in the bucket re-filled to 4500. On the next request, this algorithm would reduce the sleep value by itself since 4500/4500 is one:

decrease_value = sleep_time * 4500 / 4500

That means it doesn’t matter how immense the sleep value is, it will adjust fairly quickly to a change in workload. Good in theory, how does it perform in the simulation?

Avg retry rate:      3.07 %
Max sleep time:      17.32 seconds
Stdev Request Count: 78.44

Time to clear workload (4500 requests, starting_sleep: 1s):
84.23 seconds

This rate throttle strategy performs very well on all metrics. It is the best (or very close) to several metrics. Here’s a chart:

This strategy is the “winner” of my experiments and the algorithm that I chose to go into the platform-api gem.

My original solution

While I originally built this whole elaborate scheme to prove how my solution was optimal, I did something by accident. By following a scientific and measurement-based approach, I accidentally found a simpler solution that performed better than my original answer. Which I’m happier about, it shows that the extra effort was worth it. To “prove” what I found by observation and tinkering could be not only quantified by numbers but improved upon is fantastic.

While my original solution had some scripts and charts, this new solution has tests covering the behavior of the simulation and charting code. My initial solution was very brittle. I didn’t feel very comfortable coming back and making changes to it; this new solution and the accompanying support code is a joy to work with. My favorite part though is that now if anyone asks me, “what about trying " or "have you considered " is that I can point them at [my rate client throttling library](https://github.com/zombocom/rate_throttle_client), they have all the tools to implement their idea, test it, and report back with a swift feedback loop.

`gem 'platform-api', '~> 3.0'`

While I mostly wanted to talk about the process of writing rate-throttling code, this whole thing started from a desire to get client rate-throttling into the platform-api gem. Once I did the work to prove my solution was reasonable, we worked on a rollout strategy. We released a version of the gem in a minor bump with rate-throttling available, but with a “null” strategy that would preserve existing behavior. This release strategy allowed us to issue a warning to anyone depending on the original behavior. Then we released a major version with the rate-throttling strategy enabled by default. We did this first with “pre” release versions and then actual versions to be extra safe.

So far, the feedback has been overwhelming that no one has noticed. We didn’t cause any significant breaks or introduce any severe disfunction to any applications. If you’ve not already, I invite you to upgrade to 3.0.0+ of the platform-api gem and give it a spin. I would love to hear your feedback.

Get ahold of Richard and stay up-to-date with Ruby, Rails, and other programming related content through a subscription to his mailing list.

Schneems - Programming Practices, Performance, and Pedantry

It's dangerous to go alone, `pub` `mod` `use` this.rs

Tutorial: Naievely require/load-ing files in Rust

Watch for changes with cargo-watch

Back to the tutorial

Tutorial: Importing nested files

Calling cow_date from main

How can I load that file?

What is that mod.rs file?

Information dump

Code paths in Rust

Code paths starting with self and (unqualified)

Differences between self and (unqualified)

Code paths starting with crate

Code paths starting with super

Code path recap

Using use

Renaming paths with use

Glob imports with use

Import an extension trait with use

File/State permutations

Modules != files

What is github.com/zombocom and why most of my Ruby libraries there?

Why a custom org?

Why name it zombocom?

Pairing on Open Source

Topics

Transcript

My book 'How to Open Source' launches today!

Launch week is now over

The room where it happens: How Rails gets made

The Basecamp incident

Basecamp concerns hit Rails

How does someone get commit access to Rails?

Rails Contributors vs. Rails Core

How I GUESS someone gets on Rails Core

How do major features happen in Rails?

Explicit and Implicit Governance

Moving forwards

Migrating a Ruby Library from TravisCI to CircleCI

Nostalgia first

Why CircleCI?

Transitioning from .travis.yml to .circleci/config.yml

Break it down, start at the end

Consuming parameters

References

Version and orbs

Retro

Squash Unexpected-End errors with syntax_search

TLDR;

Back Story

The impossible goal

Relax constraints and add information: Indentation informed syntax

Search for syntax

Defining the roads

Turn-by-turn

Complicating scenarios

Next steps

Triage with Me - 11 issues & 2 PRs in 1.5 hours

Triage session 1/4

Issue 1 Heroics should use keep-alive connections #16

Issue 2 Examples produced by the CLI don’t show correctly help options #28

Triage session 2/4

Issue 3 feature: automatic generated web admin ui #36

Issue 4 Errors should go to stderr #40

Triage session 3/4

Issue 5 Escape identifiers #41

Issue 6 chunked encoding support #42

Issue 7 Validating request payload client side #51

Triage session 4/4

Issue 8 [Add connect_url or similar method to generated clients #52

Issue 9 Exposed ETags #63

Issue 10 UTF-8 with bom fails generating a client

Issue 11 I dont have definitions in my json schemas #67

(Bonus) Paring pull-request session 5/4

Wrap Up

The Life-Changing Magic of Tidying Ruby Object Allocations

Jump to:

Intro to Tidying Object Allocations

Tidying Example 1: Active Record respond_to? logic

Calling `cow_date` from main

What is that `mod.rs` file?

Code paths starting with `self` and (unqualified)

Differences between `self` and (unqualified)

Code paths starting with `crate`

Code paths starting with `super`

Using `use`

Renaming paths with `use`

Glob imports with `use`

Import an extension trait with `use`

Transitioning from `.travis.yml` to `.circleci/config.yml`

Issue 8 [Add `connect_url` or similar method to generated clients #52

Tidying Example 1: Active Record `respond_to?` logic

`gem 'platform-api', '~> 3.0'`