Protocol Buffers: beyond JSON for modern data exchange

Origins and evolution

Protocol Buffers (protobuf) were born out of Google's internal development needs in the early 2000s. Faced with the challenge of efficiently serialising structured data between different services and languages, Google's engineers built this language- and platform-independent mechanism. Released as open source in 2008, protobuf has become a cornerstone of efficient data exchange in distributed systems.

What are Protocol Buffers?

Protocol Buffers are a method for serialising structured data for use in communication protocols and data storage. Unlike text-based formats such as JSON or XML, protobuf uses a binary format, which means smaller data sizes and faster parsing. At the heart of protobuf are schema definition files (.proto) that specify the structure of the data:

syntax = "proto3";

message Person {
  string name = 1;
  int32 age = 2;
  repeated string hobbies = 3;
}

This schema-based approach offers several advantages:

1. Type safety: The schema guarantees data consistency between producers and consumers, preventing runtime errors.
2. Code generation: Types and models are created automatically in multiple languages at compile time (protobuf supports practically every language).
3. Versioning support: Built-in mechanisms to evolve data structures.

When working with protobuf, the workflow is as follows: define the .proto files → generate the code → use the generated, type-safe serialisers and deserialisers in your own language.

Note that when you define messages, you must specify a number for each property. This is the actual identifier in the binary encoding. There are several aspects to keep in mind, which you can find in the official guide.

Use cases

Protobuf excels in scenarios that call for:

• High-performance data transfers
• Strict schema validation
• Cross-language compatibility
• Optimised data storage size

Google also used Protocol Buffers as the main data transfer format for its own RPC framework. gRPC is a high-performance RPC framework, used mainly for server-to-server communication (but it can also be used in other applications, including browsers), providing:

• Strongly typed service contracts
• Efficient binary communication
• Automatic client/server code generation

One interesting aspect of gRPC is its tight integration with .proto files. You can define services, requests and responses directly in the contracts, and the protobuf compiler will generate strongly typed clients and servers for any language.

syntax = "proto3";

service TodoService {
  rpc CreateTodo (CreateTodoRequest) returns (TodoResponse);
  rpc GetTodo (GetTodoRequest) returns (TodoResponse);
  rpc ListTodos (Empty) returns (TodoListResponse);
  rpc DeleteTodo (DeleteTodoRequest) returns (Empty);
}

message Todo {
  string id = 1;
  string title = 2;
  string description = 3;
  bool completed = 4;
}

message CreateTodoRequest {
  string title = 1;
  string description = 2;
}

message GetTodoRequest {
  string id = 1;
}

message DeleteTodoRequest {
  string id = 1;
}

message TodoResponse {
  Todo todo = 1;
}

message TodoListResponse {
  repeated Todo todos = 1;
}

message Empty {}

This example shows a classic CRUD application defined entirely in a single .proto file. It will generate functions and methods usable in any supported language, to implement clients and servers directly at compile time.

Our use case: high-frequency WebSocket communication

At Quinck, we used protobuf for a unique use case: enabling high-frequency, type-safe binary WebSocket communication between browser (TypeScript) and server (Go). Since we were exchanging data at a fairly high rate (about 100 messages per second), we wanted to avoid the overhead of JSON's size and parsing performance.

The implementation is very simple: just use a standard WebSocket server and client, configure both for binary communication and generate the right types for each of them at build time from the single .proto file.

We also needed to scale the WebSockets horizontally. To do that, we decided to use Redis's pub/sub system to route messages between instances. Since Redis supports binary payloads, it can naturally carry our encoded protobuf messages safely!

Main benefits we got:

1. Bandwidth reduction: The binary format minimised data transfer (by a lot, about 70% of the original message size).
2. Type safety: End-to-end type checking prevented runtime errors.
3. Performance: Fast serialisation/deserialisation improved real-time capabilities and frontend performance.

Challenges and considerations

Although Protocol Buffers offered significant advantages in several use cases, we ran into some difficulties. protobuf's binary format made manual debugging and data inspection harder than with JSON. Also, for our use case, we had to bring in additional tools and dependencies (the protoc compiler and plugins) to generate models and deserialisers for TypeScript in the frontend and Go in the backend, unlike JSON, which is standard in both languages.

Conclusions

Protocol Buffers and gRPC have proved invaluable tools in our development stack. While JSON remains excellent for many use cases, protobuf's efficiency and type safety make it the better choice for secure, high-frequency, performance-critical applications. Our successful implementation in browser-server communication shows its versatility beyond traditional backend services.