API first - Process for internal services

FollowingFollow

Passionate about technology, music and playing guitar. A big fan of New Balance sneakers.

Overview

FARFETCH is a dynamic and agile company with constant changes - it has a large number of teams, a microservices architecture where each team is responsible for a set of services, and each service could have several consumers.

In this scenario, how can we ensure that the constant changes in services do not affect or even make consumers stop working?

With that in mind, the API first model for internal services is being adopted, with a central point where each service has an OpenAPI specification containing every description of what it provides. The idea is that every change needs to be designed and reviewed before being implemented.

Benefits

It is expected that having a single point where the process of design, validation, and review of the internal APIs are done will lead us to:

Improvements in quality and consistency;
Reducing the number of incidents due to breaking changes;
Breaking change detection and consumers notifications;
Enabling faster feedback using tools to:

Validate automatically rules, versioning, and retro compatibility;
Mock generation with OpenAPI specifications;

Increasing productivity with the adoption of code generation;
Removing the diamond dependency problem;
Having "a big picture” of all internal API’s.

How does it work?

There is a monorepo with folders split by domain areas and each team is responsible to submit their services specifications. The repository structure is described in Figure 1 below

Figure 1 - Repository Structure

Once a new specification is submitted or something changes, a pipeline runs and a validation flow starts. The validation flow is described in Figure 2 below:

Figure 2 - Validation Flow

The pipeline validates automatically:

Version: It checks if the version follows the semver convention, and also if the version was incremented. When a spec has changes, the validation tool will require the version incrementation
Rules: It checks if the endpoints, components, and properties are following the internal rules related to:

Nomenclature
Path versioning
Return types
Status codes

Changes: It validates the new version against the previously approved version and checks that the changes don’t cause any breaking changes.

Intentional breaking changes

There are situations where changes are necessary even if it will cause breaking changes to the consumers. In these cases it is possible to submit intentional changes by adding a line to the breaking-change file with the version and description of the change, you can see the complete flow in Figure 2 above.

Once a breaking change occurs, a bot will add a comment in the merge request tagging the consumers of the service so that they can become aware of the changes, and have the chance to share questions or possible issues that can affect their applications.

Validating the implementation

There is a tool that runs in the specs pipeline that can be used on the service CI pipeline to validate the implementation against the approved specification.

Merge Validation Bot

Once the specification changes have been submitted and reviewed by the service maintainers, a label is added to the merge request. A bot will automatically check if the merge request fulfils all the requirements, a rebase will be performed and then merged. If there is any error, the author and maintainers of the service will be notified directly through Slack.

Final Thought

Due to Farfetch’s scale and complexity, having a well-defined and automated process that assures the quality, coherence, and integrity of our APIs is very important. It provides a high-level vision of our API status and maturity. Together with the flexibility for a quick response to the constant changes, it offers the right balance between speed innovation and robust processes. Next iterations include improving the developer experience and leveraging this platform to other capacities like SDK generation, stubs and a better API version governance.