Last winter, as a fairly new hire, I got frustrated with our APIs. It is hard to find the right one, they only work in certain scenarios, domain internal methods are mixed in with external ones, building requests in postman required a lot of guesswork and only works for servers with that are running our http to gRPC conversion library (bifrost). We have homebuilt tools for generating API-wrapper SDKs but don’t run them for all languages or publish the output because they are buggy and require manual intervention (opposed to optional manual additions). The list of things that could be improved to make the lives of internal and external developers better goes on.
For my first hackathon project I looked into converting our proto definitions into OpenAPI files that could be loaded into Postman and documentation tools. I was not very familiar with either of the specs so I went looking for open source projects that met my needs. The closest one I found was implemented in java. I was hoping to tie it into mscli which is the tool we use for SDK generation.
Between learning mscli’s multi step docker process and releaning java on a new mac I did not make much progress. It did however open the door to conversations about how we can invest more in our API development process.
Just before we all switched to COVID mode I joined a few formal discussions about working full-ish time on APIs similar to how Branson handels building tools for i18n. That lead to a project charter. Allan already was looking into hiring a technical writer to help him provide partner facing documentation and support of our existing APIs. I was likely to join them with a focus on building new.
This week I started working full time on partner facing APIs. It is not quite the tooling for internal developers that I originally had in mind but definitely exciting.
So what am I building? I will be creating a brand new set of “Platform APIs” based on REST objects using HATEOAS, OpenAPI, and JSON:API for guidance. We choose to go back to providing REST APIs as we had received feedback from many partners that our gRPC APIs were hard to use and they required us to maintain SDKs for each language our partners develop in. We found success in describing our older Marketplace APIs using OpenAPI. It is a popular format for describing REST APIs that has a community of people building documentation tools with “Try It Now” functions and simple tools that let partners generate SDKs in over 40 languages on the fly. For the body of our requests and responses we have decided to back JSON:API. It is one of several emerging standards so we hope to see more tooling around it. I’ll follow up
Alongside these new APIs we will be releasing Architecture diagrams, a set of recipes, and a data glossary to help tie the whole picture together.
Our current process for internal APIs will not change. Protobuf over gRPC will still be the standard for internal communications.
The new REST endpoints will provide a translation layer from the way our partners think about our systems to the way we actually built them. Defining the specification for external endpoints is going to be limited to a small team so that we have a consistent tone of voice and format across all of them. Writing documentation is a prerequisite to any modifications. We will require the help of our feature teams in maintaining the translation layer.
I expect to have an RFC or two out later this week outlining my vision for the routes.
Processing Overview: The yellow parts are managed by our infrastructure teams while the green parts are managed by feature teams.
We are still trying to determine what exactly the org structure will be. At the moment I am a team of one with lots of collaboration with from our CTO. Holding a retro meeting at the end of each sprint does not make a lot of sense. What I am going to try doing instead is writing blog posts. That will give me an opportunity to reflect and plan for the future. It will also make my work visible to others so that I am not working in secret.
I hope you enjoy following along in my journey.
