Skip to content
Domain Specific Language

A four-hour rabbit hole into (mostly) generated Rust code

There are no official Rust libraries for Google APIs available yet, but there are some (mostly) auto generated ones that are fine to use albeit a bit verbose.

I've previously used google-apis-rs successfully when I needed to read Secret Manager secrets for a side project.

Hot on the heels of Thoughts on ethical software I decided to convert the same project from using MySQL to the snazzily named Firestore in Datastore mode.

Let's go down the rabbit 🐇 hole 🕳.

An error appears

His terminal cursor blinked with the cold determination of an experienced murderer, many years since his first kill, but never caught.

Reqwest Error: error decoding response body: invalid base64 input at line 3 column 1

Just kidding. This is not a murder story. I'm not kidding about the error though.

Some code to start things off.

Some of the code below, like crate::accesstoken::GoogleToken isn't available publicly, but it's based on gcp_auth = "0.1.4". If you want, contact me, and I'll probably send you the code.

[dependencies]
google-datastore1 = { git = "https://github.com/google-apis-rs/generated" }
use crate::accesstoken::GoogleToken;
use eyre::{Result, WrapErr};
use google_datastore1::Client;
use google_datastore1::schemas::{BeginTransactionRequest, BeginTransactionResponse};

pub fn datastore_transaction_test() -> Result<()> {
    let gtoken = GoogleToken::get_token().wrap_err_with(|| "Could not get google token")?;
    let client = Client::new(gtoken);
    let projects = client.projects();
    let builder = projects.begin_transaction(
        BeginTransactionRequest {
            transaction_options: None,
        },
        "some-google-cloud-project-name",
    );
    let transaction: BeginTransactionResponse = builder.execute()?;

    // ...

    Ok(())
}

Where to start?

Soul smashing. I only have a few precious hours per day to code, and now I have a decision to make. Do I give up? Do I find another library?

Resolve and dedication is the name of the game. Plus most of the time it's something silly like making the wrong assumptions about the library I'm trying to use. E.g. is my authentication token correct? Did I give the wrong project name?

Is the request good?

I'm using CLion which includes Rust debugging. The first thing I want to know is, am I making an error in my request?

Let's step into builder.execute() and see what it is hiding.

pub fn execute<T>(self) -> Result<T, crate::Error>
where
    T: ::serde::de::DeserializeOwned + ::google_field_selector::FieldSelector,
{
    let fields = ::google_field_selector::to_string::<T>();
    let fields: Option<String> = if fields.is_empty() {
        None
    } else {
        Some(fields)
    };
    self.execute_with_fields(fields)
}

It's just collecting some parameters and calling execute_with_fields

pub fn execute_with_fields<T, F>(mut self, fields: Option<F>) -> Result<T, crate::Error>
where
    T: ::serde::de::DeserializeOwned,
    F: Into<String>,
{
    self.fields = fields.map(Into::into);
    self._execute()
}

Same story, let's go into _execute

fn _execute<T>(&mut self) -> Result<T, crate::Error>
where
    T: ::serde::de::DeserializeOwned,
{
    let req = self._request(&self._path())?;
    let req = req.json(&self.request);
    Ok(crate::error_from_response(req.send()?)?.json()?)
}

crate::error_from_response(req.send()?)?.json()? looks interesting! .json() specifically sound like something that takes a string (or something) and gives us json (of type ::serde::de::DeserializeOwned).

.json() takes us out of the google-datastore1 crate and into reqwest, and the method looks like this:

pub fn json<T: DeserializeOwned>(self) -> crate::Result<T> {
    wait::timeout(self.inner.json(), self.timeout).map_err(|e| match e {
        wait::Waited::TimedOut(e) => crate::error::decode(e),
        wait::Waited::Inner(e) => e,
    })
}

Cool. self.inner.json() looks juicy. Let's dig into that.

pub async fn json<T: DeserializeOwned>(self) -> crate::Result<T> {
    let full = self.bytes().await?;

    serde_json::from_slice(&full).map_err(crate::error::decode)
}

This 👆 is the sixth code block in this blog post, and let's be honest you probably didn't read any of them. But this is where the fun starts, so pay attention. self.bytes() returns a type called Result<Bytes> which I'm guessing are going to be bytes :).

What I want to do in this section is to see if the response from Cloud Datastore is malformed. I want to know where that error message is coming from, and so far I haven't got a clue.

I'll place a breakpoint on the last line in the method using CLion's built in LLVM debugger.

Now the first time the breakpoint is hit, in my case it's just my GoogleToken being deserialized. I press Resume Program and wait for the second one.

I just want to know what's in the &full reference. I could do that in a couple of ways. The first one is to just change the code right there to print out &full on the command line. Let's try that.

Now the method looks like this

    pub async fn json<T: DeserializeOwned>(self) -> crate::Result<T> {
        let full = self.bytes().await?;

        println!("{:#?}", String::from_utf8(full.to_vec()));
        serde_json::from_slice(&full).map_err(crate::error::decode)
    }

Unfortunately CLion doesn't pick up on this change automatically, I have to run cargo clean first. If it compiles, and I want to make a change, I have to run cargo clean again. Maybe there's a better way? I wonder while waiting for everything to compile. I wait. For a long time. Finally my terminal comes to life.

"{\  \"transaction\": \"Ed4MD3Hb1vERIlkA3ModsqJ/tO7aMMgajDXq<MORE_STUFF...>==\"\n}\n"

Great! The API call works. But before continuing, I have to know - can this be made easier? Fortunately, it can! We can use the Memory View feature in CLion. Just expand full in the Variables view, then right click on ptr and select Show in Memory View:

And there it is, hiding in plain sight, my transaction in glorious hexadecimal but also in inglorious text.

Where is the error?

I know that the value in the json() method should be at least readable by serde, and I think that transaction looks like valid Base64, at least it's got the signature Base64 == padding on the end.

I run the Base64 through my trusty terminal

test% pbpaste | base64 -D
����O_a�0ۉΗ��j�y�ϵ��\]�Ϧ�;A�
`@@Yc�ԏ�Y�f�C��с(6���iR��%

Looks legit :). What happens if I throw in an invalid character like *?

test% pbpaste | base64 -D
Invalid character in input stream.

My transaction is probably OK.

I continue stepping into serde_json::from_slice and from there into from_trait(read::SliceRead::new(v)). It's just generic methods and macros all the way down, and honestly not many things to use to make sense of what might be happening.

I know that serde is trying to deserialize JSON and I know that Rust can pick implementations based on the return value of a method. I probably need to go back to where it all started.

let transaction: BeginTransactionResponse = builder.execute()?;

Let's see what BeginTransactionResponse is hiding!

pub struct BeginTransactionResponse {
    #[doc = "The transaction identifier (always present)."]
    #[serde(
        rename = "transaction",
        default,
        skip_serializing_if = "std::option::Option::is_none"
    )]
    pub transaction: ::std::option::Option<::google_api_bytes::Bytes>,
}

What's this Bytes thing? Go to Declaration, please.

#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash)]
pub struct Bytes(pub Vec<u8>);

It's a newtype over Vec<u8>. Clicking the gutter icon (green "I") might help.

There's an implementation for Deserialize<'de> for Bytes. Let's check it out.

impl<'de> Deserialize<'de> for Bytes {
    fn deserialize<D>(deserializer: D) -> ::std::result::Result<Bytes, D::Error>
    where
        D: Deserializer<'de>,
    {
        let encoded = String::deserialize(deserializer)?;
        let decoded = BASE64_CFG
            .decode(&encoded)
            .map_err(|_| ::serde::de::Error::custom("invalid base64 input"))?;
        Ok(Bytes(decoded))
    }
}

That error looks familiar. If you didn't already guess it, I'm looking straight at that BASE64_CFG and wondering what that does.

use radix64::URL_SAFE as BASE64_CFG;

And radix64::URL_SAFE is...

/// Encode and Decode using the URL safe characer set with padding.
///
/// See [RFC 4648](https://tools.ietf.org/html/rfc4648#section-5)
pub const URL_SAFE: UrlSafe = UrlSafe;

Is it possible that some Google APIs use UrlSafe Base64 encoding, and others don't? It's impossible to tell from the documentation. Some APIs do seem to specifically say "Use URL safe Base64", but most don't.

I once again decide to edit a file. Yeehaw. Change use radix64::URL_SAFE as BASE64_CFG; to use radix64::STD as BASE64_CFG;. Recompile.

It works! No more errors.

Be a good OSS citizen

The least I can do is to report the issue. Hopefully the repository is still maintained, and hopefully the maintainers appreciate a bug report. Said and done, the bug report is here. https://github.com/google-apis-rs/generated/issues/3. And a fair bit shorter than this blog post.