Macros and teams

One afternoon I decided that I wanted to learn a bit about the details of HTTP based APIs and user authentication. It's something I've never done from scratch. It looked like a cool little side research. While wiring the persistence I discovered that I wanted to simplify my testing armoring.

Which led to my first non-trivial proc macro.

What in turn made look back and reflect on how teams I've work with have interacted with this particular feature of rust. What I perceive as their real costs and how, at the end, "hoomans human".

DIY Authentication

This story begins with an article I found in shuttle.rs's documentation about authentication. I've written quite some backend lately but I never had to care about authing agents. It was a good opportunity to learn about it by implementing something from scratch.

Shuttle's article covers a lot of ground. I believe it's a reasonable starting point. I wasn't particularly interested in hosting this app inside shuttle.rs so I dissected the sample and started writing the glue myself. Their proposed implementation looks a bit like:

1. Postgres on persistence

2. sqlx glueing rust to the DB

3. Axum serving the calls

4. Templated server-side rendering for the pages

It's a perfectly reasonable stack if you're an edge-cloudy provider and part of your goal is to sell services to the reader. So one of my first calls was to yeet Postgres and use the borderline almighty SQLite.

Notes on SQLx

sqlx is an async library that let's you connect and query SQL DB's, among others SQLite and PostgreSQL. It plays well with axum. It includes some tooling, like the SQLx CLI, with operations such as:

• create a new DB
• runs migrations
• enables static analysis of your queries

This is a good project that I've used a couple times and it always delivers. I had very few surprises with it.

TDD-ing + SQLite: the problem

After a bit of reading and github spelunking I was ready to write some tests for my totally tutorial based code. Very soon I found myself writing the following block again and again:

use authentication::entities::user::User;
use authentication::persistence;

const PATH_DIR_MIGRATION: &str = "./migrations";

#[tokio::test]
async fn sqlite_users_can_persist() {
    // Scaffolding
    let sqlite_pool = test_sqlite::sqlite_create(
        PATH_DIR_MIGRATION,
        "./tests_output/sqlite_users_can_persist.sqlite",
        32,
    )
    .await
    .expect("test scaffolding DB should be possible to create");

    // this is the core of the test
    let foo_user = User::new("foo-name");

    let _new_user = persistence::save_user(&sqlite_pool, &foo_user, "foo-password")
                    .await
                    .expect("This DB was empty, it should be possible to create a new user!");
    // more scaffolding
    sqlite_pool.close().await;
}

I hope the code is clear but in summary, what I'm doing is:

1. create a SQLite and run the current migration
2. my test
3. close the pool

In case you want to get properly bored you can read about the details of test_sqlite::sqlite_create below, in a handy appendix. Or in a github gist.

There's a lot of scaffolding.

Maybe it's possible to reduce all this noise?

Is it perhaps a good candidate to use macros and reduce the cruft?

Rust Macros

When approaching rust it's somewhat tempting to read the word "macro", shortcut into #DEFINE, and skip the whole thing³. At least this is what I did in my first approach. Perhaps it's a naming problem. Buried in the manual itself we find a way better word: metaprogramming.

Fundamentally, macros are a way of writing code that writes other code, which is known as metaprogramming.

Then, the manual adds:

Metaprogramming is useful for reducing the amount of code you have to write and maintain, which is also one of the roles of functions.

Around me, the first real contact with macros any new rust developer would experience happens when working with serialization and deserialization, usually through serde. The always informative @jonhoo has a wonderful piece on this crate:

Through serde we get an intuition on how powerful this feature can be.

TDD-ing + SQLite: take 2

In previous section I was complaining about the cruft I was copy-paste-ing over and over. But, what if the test could look like this instead?

#[test_sqlite]
async fn sqlite_users_can_persist(sqlite_pool: &sqlx::SqlitePool) {
    let foo_user = User::new("foo-name");

    let _new_user = persistence::save_user(sqlite_pool, &foo_user, "foo-password")
                    .await
                    .expect("This DB was empty, it should be possible to create a new user!");
}

Where all the scaffolding is hidden away inside the macro? Let's explore a potential implementation.

#[test_sqlite] macro

It's important to note that currently (March 2024) you need to follow a strict method if you want to develop a procedural macro. You'll need to add a new crate, name things appropriately and import a couple of low level crates to your sources. It's a bit of setup.

Let's keep in mind that the following macro code is a sample and that I'm not exactly a hardcore crustacean. I know it works, and I know it's keeping my code cleaner, I know it has a bunch of redundant clones in it. It's also a reasonable "non trivial" sample.

extern crate proc_macro;
use quote::{format_ident, quote, ToTokens};
use syn::{ItemFn, LitStr};

#[proc_macro_attribute]
pub fn test_sqlite(
    input: proc_macro::TokenStream,
    item: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
    impl_test_sqlite(input.into(), item.into()).into()
}

// Notice the "proc_macro2" TokenStream
fn impl_test_sqlite(
    attr: proc_macro2::TokenStream,
    item: proc_macro2::TokenStream,
) -> proc_macro2::TokenStream {
    // Supporting the feature #[test_sqlite("../path/to/your/migration")]
    //     or default to "./migrations"
    let migrations_dir = if attr.is_empty() {
        "./migrations".to_string()
    } else {
        // Try to parse a String Literal -> LitStr
        let t: LitStr = match syn::parse2(attr.into_token_stream()) {
            Ok(i) => i,
            Err(e) => return e.into_compile_error(),
        };
        t.value()
    };

    // Try to parse an item that's a function -> ItemFn
    let mut test_body: ItemFn = match syn::parse2(item.clone()) {
        Ok(item) => item,
        Err(e) => return e.into_compile_error(),
    };

    let original_ident = test_body.sig.ident.clone();

    // Change the name of the original function by suffixing `_impl`
    test_body.sig.ident = format_ident!("{}_impl", test_body.sig.ident);

    let test_body_ident = test_body.sig.ident.clone();

    quote! {
        #[::tokio::test(flavor = "multi_thread")]
        async fn #original_ident () {

            #test_body

            let test_output_path = format!("./tests_output/{}.sqlite", stringify!(#original_ident));
            let pool = ::test_sqlite::sqlite_create(
                #migrations_dir,
                &test_output_path,
                4,
            )
            .await
            .expect("test scaffolding DB should be possible to create");

            #test_body_ident(&pool).await;

            pool.close().await;
        }
    }
}

If you prefer, there's a version hosted in gist.

Through the wonderful rust-analyzer we can recursively expand and check what's reaching the compiler:

#[::core::prelude::v1::test]
fn sqlite_users_can_persist() {
    let body = async {
        async fn sqlite_users_can_persist_impl(sqlite_pool: &sqlx::SqlitePool) {
            let foo_user = User::new("foo-name");
            let _new_user = persistence::save_user(sqlite_pool, &foo_user, "foo-password")
                    .await
                    .expect("This DB was empty, it should be possible to create a new user!");
        }
        let test_output_path = format!(
            "./tests_output/{}.sqlite",
            stringify!(sqlite_users_can_persist)
        );
        let pool = ::test_sqlite::sqlite_create("./migrations", &test_output_path, 4)
            .await
            .expect("test scaffolding DB should be possible to create");
        sqlite_users_can_persist_impl(&pool).await;
        pool.close().await;
    };
    tokio::pin!(body);
    let body: ::core::pin::Pin<&mut dyn ::core::future::Future<Output = ()>> = body;
    #[allow(clippy::expect_used, clippy::diverging_sub_expression)]
    {
        return tokio::runtime::Builder::new_multi_thread()
            .enable_all()
            .build()
            .expect("Failed building the Runtime")
            .block_on(body);
    }
}

If you look carefully you'll see how macros are operating recursively. In particular how test_sqlite embeds a call to tokio::test. The compiler will see the full expanded code and if something fishy happens it'll complain as expected. Assuming you're using a "plebs editor"⁴ (ie. VSCode) this effectively means that you'll see a squiggly reporting an issue on code that you can't see.⁵

The team cost of macros

In the previous sections I described what me-myself wanted to solve: reduce the cruft in these tests and the way I decided to go ahead with it: add a procedural macro. Working alone this is a simple step for the codebase: I know the implementation in-n-out, I wrote it, hardened it, tested it. And more importantly, I don't need to convince anyone to adopt this solution.

But let's imagine a team that's starting with rust or perhaps, for reasons, hasn't had the need to develop their own macros before. What is this team committing to?

1. A new crate for the macro. Usually a trivial matter of fixing a bunch of Cargo.toml entries. But let's not forget our beloved build engineers and their fancy robots. They'll have to be sure to wire everything.

2. Introduced the dependencies: syn, quote and proc-macro2. Probably a minor issue since these dependencies are constrained to the crate containing the macros.

3. Rust as TokenStream. When writing macros our goal is to transform the input into valid rust code. This requires a bit of mental gymnastics on which a developer needs to shift its focus from the business domain into a way more formal language domain.

4. The quote! blocks are effectively a DSL. The sample above is not exactly trivial but it's on the very simple side. quote! blocks can get complicated and difficult to parse if something bad happens in the macro. Or if you need to go back and extend its features.

5. Behind the scene code. The code the team writes looks different than the code reaching the compiler.

6. Using new features of the tooling: rust-analyzer expand macro operation, or perhaps the cargo-expand. I haven't been able to naturally integrate macros with my day to day IDE experience⁶. A problem that makes the code feel distant, obscure and perhaps even scary.

Those being the "hard aspects" of macro maintenance in a team. I believe there's also a policy aspect. Let's be cheeky and call it ...

Macro "sanity spectrum"

As with many programming language features (where C++ templates come to mind) there seems to be a spectrum of adoption for macros. A gradient that starts in: "this feature is never used" and goes all the way up to "arcane black magic".

It's quite difficult to "not use macros at all" when using rust. Many are borderline mandatory. An example from a simple async CLI:

#[tokio::main]
async fn main() -> Result<()> {
    println!("You probably want to RUST_LOG=info or lower");
    // [...]
}

Where the first lines of that async CLI is using 2 macros already. You could expand all that "by hand" and have pure rust in your code but it feels quite silly. They're a tremendous feature.

The potential problem happens "at the other side of the spectrum" where macros are doing a hell of a lot. Perhaps you're building UI and targeting web. You might be interested in using rust-as-react through yew (it's a magnificent library, btw) And, since you're a modern fella, you don't want to use filthy structs, you want to use the chaddest function components! You're in luck, the #function_component macro has you covered. And then your code will look a lot like javascript.

The human side

We can say that teams are a bunch of guys working together towards a shared goal. They'll bring their own preferences and skill sets to the collective. And in the case of software developers this includes strong opinions on abstraction and tolerance for consuming piles of documentation and scaffolding.

In relation with macro adoption I had the chance to meet both extremes. Those that consider the costs as being too prohibitive. The others that see the opportunity to extend rust through macros effectively making it look as other languages.

Wrapping up

I perceive a difference between macros and other features of rust. They seem to come with a steep price tag for the adopting teams. At the same time they open a fantastic array of possibilities that go well beyond the very simple example contained in this article.

Where I see the real issue, in the context of teams, is in the hoomans. Unless you're careful navigating the "sanity spectrum" there's a real chance of leaving some of your devs out and behind. And it's easy to reach narratives where some of your team colleagues will describe your carefully tailored macros as "over complex" or "unnecessarily flashy" while others embrace their usefulness and use them for absolutely everything.

Good hunt out there!

Appendix: sqlite_create

For my own future reference and in case you're curious about the details of sqlite_create it looks like this:

use std::process::Command;

use anyhow::Result;

use sqlx::{sqlite::SqlitePoolOptions, Pool, Sqlite};

async fn sqlite_migrate_as(migration_dir: &str, path_new_db: &str) -> Result<String> {
    let cs = format!("sqlite:{}?mode=rwc", path_new_db);

    let output = Command::new("cargo")
        .arg("sqlx")
        .arg("migrate")
        .arg("run")
        .arg("--source")
        .arg(migration_dir)
        .arg("--database-url")
        .arg(&cs)
        .output()?;

    if !output.status.success() {
        let stderr = std::str::from_utf8(output.stderr.as_slice())?.to_string();
        return Err(anyhow::anyhow!(stderr));
    }

    Ok(cs)
}

pub async fn sqlite_create(
    migration_dir: &str,
    path_new_db: &str,
    max_connections: u32,
) -> Result<Pool<Sqlite>> {
    let p_output_db = std::path::Path::new(path_new_db);
    if p_output_db.exists() && p_output_db.is_file() {
        std::fs::remove_file(p_output_db)?;
    }

    let p_migration_dir = std::path::Path::new(migration_dir);
    if !p_migration_dir.exists() || !p_migration_dir.is_dir() {
        panic!("Specified migration dir: `{}` not found", migration_dir);
    }

    let cs = sqlite_migrate_as(migration_dir, path_new_db).await?;

    Ok(SqlitePoolOptions::new()
        .max_connections(max_connections)
        .connect(&cs)
        .await?)
}

Where I'm using SQLx CLI (through cargo).

The core concept here is: create this SQLite DB and run this migration, if something fishy happens go BANANAS.