Documentation Framework
"Every engineer is also a writer"*
As businesses grow, so does the need for technical documentation. Up-to-date technical documentation can save precious time to developers, and, consequently, save money to our company. The production of up-to-date technical documentation should include not only technical writers but also developers (engineers). Technical writers know how to write, but developers are the subject matter experts.
Instead of doing all the writing, technical writers should work side-by-side with teams, providing them the means to produce their own documentation. In this article, I will share how we've integrated developer's use of Git with our technical documentation publishing process in Confluence. I will also outline our Documentation Framework, which organizes and standardizes pages and templates to produce up-to-date code documentation at scale.
Personal journey
When I first started working at FARFETCH, back in May 2017, my job was to write documentation for FARFETCH Public API in the Technical Writing Team (TWT). Back then, the TWT managed an internal site for developers: the FF Developers site (FFDevelopers). FFDevelopers was mainly focused on the documentation of FARFETCH Public APIs and it could be accessed by brands, boutiques, and other FARFETCH Partners, developing their own software to interact with FARFETCH Platform. Documentation sources for the FFDevelopers were written in DITA and versioned in a Git repository. The site was generated using the DITA Tool-kit.
As shown in Figure 1, FFDevelopers is structured in layers according to DITA types of topics (pages): the concept layer, the tutorial layer, and the reference layer. The concept layer briefly describes business and API concepts related to the public APIs. The tutorial layer contains a set of step-by-step "how-to” pages explaining how to implement some of the core functionality of FARFETCH Platform. The reference layer contains detailed descriptions of the public API resources generated from Open API specifications that are managed both by developers and members of the TWT.
As mentioned before, the FFDevelopers portal was focused, mainly, on documenting FARFETCH public API. Because it was also available for FARFETCH Partners, it followed a strict style guide and it had a well defined structure (Figure 1). If a developer wanted to contribute to the FFDevelopers portal by writing a tutorial, for example, this developer would have to meet the following requirements:
- Learn DITA.
- Learn how to apply the style guides defined by the TWT.
- Work in collaboration with a member of the TWT to be able to introduce the tutorial to the already established information structure of the FFDevelopers Portal.
As a result of these requirements, documentation for FFDevelopers was almost exclusively produced by members of the TWT. The only contribution that developers made was through the Open API specs of their team API.
Moreover, to register the main events of their team's daily life, developers also write what I like to call "administrative documentation”, such as meeting notes, decision logs, etc. But they felt the need to also have user-facing technical documentation such as how-to guides. The main reason to also have this type of documentation is to drastically reduce the time spent giving support to internal users of services and applications. As an example, documentation for authentication and authorization had more than 1000 viewers with an average of 3 visits per viewer. If viewers had contacted a developer every time they needed information, these numbers represented almost 3000 interruptions that developers avoided.
To produce documentation, developers mostly use FARFETCH Confluence where each team can create their own area and organize documentation as they see fit. As a result, they often mix administrative and technical documentation in the same place, but this practice is prone to major chaos!
Documentation in Confluence is difficult to manage. Mixing user-facing technical content with administrative documentation makes it confusing, hard to browse and organize. Also, developers like to have documentation next to the code and that cannot be done with Confluence pages.
By the end of 2019, I was given a new challenge: working with the FARFETCH Identity team (FFID) to document FARFETCH authentication and authorization. I moved to the Lionesa office and started working with the FFID team. By then, there were two main sites for documentation: the FFDevelopers Portal, that was focused on public APIs and managed by the TWT; and Confluence, where developers were writing their documentation, but not happy with it.
As I was not going to document the FARFETCH Public API, I faced a two-part challenge: (1) Learn enough about authentication and authorization to be able to explain it; and (2) Define the writing and publication process.
Challenge #1: Developers from the FFID team did an incredible and patient job explaining everything to me. But because authentication and authorization are such specific and technical areas, I had to research and learn the theoretical concepts behind it, so the team gave me several RFCs to read and I also attended internal and external workshops about security.
It was an incredible experience for me, because I really love to learn. What I love the most about being a technical writer is the constant need for studying and learning in order to be able to explain to others. You cannot write about what you don’t know… at least, I can’t! However, if the FFID team already had their own documentation, it would save me time that I could spend searching for information online.
After learning as much as I could in the time that I had available, I started documenting authentication and authorization in general,and the specifics of our FARFETCH implementation. I analyzed all the information and came to the conclusion that, for the first draft, I was going to have three types of information:
- "How-to guides” that would explain how to implement authentication flows.
- Manually written reference information that would explain in detail Identity endpoints and data structures.
- Automatically generated reference information that was collected from endpoint execution and needed to be presented to the user in a user-friendly way.
Challenge #2: To define the process for writing and publishing, I had to consider the available technology. The chosen solution would have to allow the automatic generation of pages and it would have to be easy enough, with a steep learning curve so that developers from the FFID team could also maintain the documentation. The available options were:
Figure 2
Given the types of information to consider and the constraints mentioned in the table above, the almost obvious choice was to use markdown. So I created a Git repository to version the documentation and hosted it on an internal site: the Documentation Hub. Then, I started structuring the documentation and writing scripts to automatically generate some of the reference documentation, such as the list of roles, list of scopes, and open id configuration. I also added other relevant information that could be fetched from endpoints or code. After having put these in place, I started writing with the help of the developers from the FFID team.
With time and the evolution of the documentation, we (me and the FFID team) came to the conclusion that Confluence is the place where FARFETCH developers search for information by default. It would be beneficial to be able to publish this documentation on Confluence as well, and this is how the Documentation Framework (DocFW) was born.
The Documentation Framework
DocFW is a framework that enables anyone to write consumer-facing technical documentation with a common format and to automate its publication. It was created from the developers' need to have an easy documentation process that wouldn't be too different from the coding lifecycle. It includes:
- A set of scripts that process markdown files and convert them to Confluence HTML.
- A set of page and document templates that help writers structure their documentation in a standardized way.
DocFW architecture
The DocFW has three stages:
- Sourcing: During this stage, writers use a Git branch to write their documents, commit them and discuss changes with other people involved in the writing process (subject matter experts, stakeholders, end users, etc). They can also test the documents with the end-users. The repository where developers version their documents is the same as the repository where they version the code. This practice aligns the documentation version with the code version.
- Processing: After finishing writing, developers request the merge of the working branch to the master branch. When the merge is approved, the pipeline runs and performs two tasks: generation of a swagger.json file with the specification of the API being merged to the master branch; and the transformation of the whole document in HTML compatible with Confluence. For both tasks, it uses the last version of the scripts available in the Git repository of the DocFW. If the service or application doesn’t have an API, it skips this task.
- Publication: In this stage, DocFW scripts use the Confluence API to compare the structure of the document in Confluence with the structure of the document in the Git repository, and then it processes pages as follows:
- Deletes pages that exist in the Confluence document, but don’t exist in the document stored in the Git repository;
- Creates pages that don’t exist in the Confluence document, but exist in the document stored in the Git repository;
- Updates pages that exist in both documents and that have been changed in the Git repository document.
Changes that were directly made in Confluence are overwritten.
Document folder
To be able to use the DocFW, each document must be in a folder that includes, at least, the following three files:
- _sidebar.md: That contains the structure of the document.
- README.md: That holds the content of the document root page.
- configuration.json: That contains information about the document, such as space where it is going to be published, labels, confluence editor version, etc.
Figure 4: Document configuration file.
When the DocFW executes, it reads the configuration file and publishes the document according to the parameters specified in this file.
Document structure
What is a document? From the point of view of the DocFW, a document is a structured set of pages with the following sections:
- Root page: It’s the main page of the document and should have some information about the document such as: who’s the target audience, how the document its structured and how it should be read, contact person or team, how to authenticate, etc
- What-is: This section should explain theoretical concepts and processes about the application or API described in the document.
- How-to: This section should have step by step "how-to” guides teaching the user how to perform relevant tasks with the API or application.
- Reference: This section should have detailed technical information like description of endpoints and respective parameters.
The root page is always mandatory, but the rest of the sections may be omitted. It all depends on the purpose of the document. For example, a document that has the purpose of only explaining concepts may not include the How-to and Reference sections.
Figure 4 shows a markdown file containing the list that defines an example of a sidebar menu with the structure of the document. This sidebar belongs to the documentation of the Pet Store API that we wrote as a toy example to exemplify the DocFW.
The first line is the root page. DocFW will create this page in a Confluence space and all the other pages as children of the root page. This page will explain who is the target audience of this document and how you can get access to the Pet Store. One of the DocFW best practices is that the file that contains the root page has the name README.md, because applications like GitLab and GitHub display README.md by default.
API Overview, in line 2, is the What-is section of the document where we explain business concepts like what is a Pet Store, what are its main features, what are Pets and some requirements for buying a Pet. Other relevant concepts related to the Pet Store are also included.
How to Buy a Pet, in line 3, is the How-to section containing step-by-step instructions on how to perform important operations of the Pet Store: create, delete, and sell Pets. Finally, the Pet Store API, in line 8, is the Reference section. It contains the specification of the Pet Store API that is displayed in Confluence using the Open API macro. The specification of the Pet Store API contains all the technical details about the API.
Page templates
DocFW also includes templates that define the structure of the different types of pages. All templates must have a title and a short description. The title is the name of the page. The short description is a small paragraph after the title and should give a glimpse of the content of the page. The purpose of the short description is to help the reader to decide if a page contains the information that the reader is looking for or if the reader should look elsewhere. DocFW has available the following templates:
- Root page: This template includes sections to define the audience and to describe the purpose of the document. It can also include a section for conventions and how to give feedback.
- How to page: This template should be used to describe step-by-step tasks and contains the following sections: a before you start section listing the prerequisites to execute the task; an overview with information that helps the user to better understand the task; a step section listing the necessary steps to perform the task; and a results section describing what the user should accomplish after performing all the steps.
- Reference page: This template should be used to describe in detail endpoints, scripts, functions, data structures, graphical components, etc. It contains a section to describe the element in detail and one or more tables to list and describe its parameters or fields.
- Navigation page: A navigation page is a page that has children. The example of Figure 5 has a navigation page with title "How to Buy a Pet” in line 3. Navigation pages include a table of links to the children pages and can have a description with additional information about the children pages.
Templates include tags to define the different types of page structure. Tags also allow automatic processing of the pages. Standardization of document and page structure helps readers to find information and it helps writers to write. Instead of structuring documents from scratch, writers can start with the recommended structure distributing information between the different sections. It also promotes writing by example. Writers can start a new document using an already written one as a guideline, instead of having to read and learn tedious style guides.
Figure 5 shows the template for a navigation page. Tags <!--title:start--> and <!--title:end--> define the beginning and end of the title and tags <!--shortdesc:start--> and <!--shortdesc:end--> define the beginning and end of the short description. DocFW uses these two pairs of tags to generate the children page table. After generation, DocFW inserts the children table between tags <!--tree:start--> and <!--tree:end-->.
Figure 6: Example of a navigation page.
Docs-as-code mindset
DocFW approach to documentation allowed writers, specially developers, to have the docs-as-code mindset. This mindset allows people who write to integrate the development of technical documentation in the code development process keeping both aligned.
In order to get people engaged to use DocFW, we (the DocFW team that includes two people: 1 technical writer and 1 developer) keep nudging them. First, we used the worth-of-mouth approach, engaging in personal contact with key teams. We started by explaining DocFW and presenting some analytics that show how much time a good documentation can save them. Although we continue to contact team members on a weekly basis, there are too many of them (more than 3000) for the number of existing technical writers in the DocFW team (just one!). Moreover, new people (developers, testers, product managers, etc) keep onboarding every month and it’s almost impossible for one person, even two, to reach them all.
Another approach that we used was to find a set of anchor pages that we included in the same Confluence space where the user guide of the DocFW was published. We borrow this idea from anchor stores (also known as anchor tenants) in malls. Anchor pages are pages that can be automatically generated, have almost daily updates, and have information useful to FARFETCH tech people in general.
We found three pages that met these criteria: the list of internal applications using the APIs, the list of internal roles, and the list of internal scopes. These three pages became the most visited pages of our documentation. In addition to exposing key information, these three pages made tech people also visit the DocFW user guide and allowed the announcement of changes to the documentation every other week nudging teams.
As a consequence of these nudges, teams started reaching out, because they wanted to start documenting their services. Right now, we have a project running as a proof-of-concept where we are experimenting with the generation of documentation from code, and DocFW is responsible for more than 1000 pages in Confluence.
Benefits and side-effects
As shown in Figure 6, using markdown to write documents and moving the source of the documentation to Git repositories allows its publication in multiple places and tech people are no longer limited to Confluence. In the case of the internal APIs, the version of the documentation is always aligned with the version of the code.
Using the templates and recommended document structure allows the normalization of the documents making it easier to read and search. In the future, if FARFETCH decides to centralize documentation in one place, we can easily track Git repositories that are using DocFW.
Moreover, documentation sites can be organized as collections of documents. If a team wants to contribute with a document, this document can be added to the site without disrupting the underlying structure of the site. It is like building the documentation site from bottom to top using documents as its building blocks. This is an important feature for the fast changing development environment we have.
Having teams producing user-facing documentation also benefits the TWT. In order to produce well structured documentation for partners, the TWT members would have to meet with the teams through several meetings in order to learn about their APIs. Now they can read it first hand then go to meetings much more informed and ready to ask difficult questions.
However, in my opinion, the most important benefit of the DocFW is the changing in the culture and mindset of tech people, and providing the means for every engineer to also be a writer.
Conclusion
DocFW was designed with the aim of overcoming some of Confluence's limitations. However, it outgrew itself and became a useful framework that can be used by all the engineers at FARFETCH. It automates more of the generation and publication of technical documentation, and it has helped to evolve the existing mindset.
When using the templates, both engineers and technical writers can produce technical documentation with a similar format. Page templates help writers to structure information as if they were writing pieces of code. Document templates make page navigation easier, because they are more predictable for the reader.
Automating the generation of some of the pages exposes information that otherwise would have to be provided by a developer. This is important because it saves writers from tedious work, and makes the information available not only to developers, but also product managers, marketing teams, among others.
We will continue developing DocFW by exploring a feature that will allow us to convert existing documentation in Confluence to markdown. DocFW is making the production of technical documentation available to everyone at FARFETCH.
If your company is starting to feel the need for technical documentation because it’s growing fast and teams are no longer working in the same physical space; or it went remote during the pandemic and remains that way; or because it’s a big company, you never thought about having technical documentation, and your developers are spending precious time explaining the same thing over and over again, you can change the mindset of your company and improve its processes with DocFW.
Having up-to-date technical documentation will help not only your developers but also your product managers and customers.
*Source: https://developers.google.com/tech-writing
