The Internet has evolved in the last 2 decades a lot. HTTP/1.1 was not enough so we have HTTP2 now. The specification we used to transfer data between the client and the server has also evolved. From XML to JSON, now we have Protocol Buffer, which is a binary spec. Let’s dive deeper.
The picture below from Wikipedia shows how exponential data is growing.
Enough HTTP2 to know gRPC
During the early days of the web, we only had static content, hardly text files, and HTML files. After then, the web grew a bit and the web server and clients started to deal with rich media like rich texts, images, and videos. Servers also started to serve dynamic content based on what clients request.
This was the start of the RPC era. A client would hit a certain endpoint with certain data, and the server’s work was to respond to that request. In the early time of RPC, XML was extensively used. We still use XML in some systems. But most of the world has moved to JSON as a communication format between the server and the client.
Now, over time, everything has evolved. One of the things which evolved is HTTP spec. Now we have HTTP2. HTTP2 is the base of gRPC. Certain properties make HTTP2 a nurturing ground for gRPC.
|Header is plaintext and not compressed||Header is compressed and binary|
|Spawns a new TCP connection on each request||uses already existing TCP connection|
|TLS is not required||Requires TLS by default, enhanced security|
Let’s learn about Protocol Buffer first
We have seen how HTTP2 is essential for gRPC, now let’s see where Protocol Buffer stands.
Although you can use gRPC with JSON, Protocol Buffer brings new things to the table. We saw that for a long time, JSON was used to send data back and forth between servers and clients. Now, what has happened is that we have taken yet another step to move from JSON to its successor.
Protocol buffers are building block of gRPC and is a replacement for JSON. Protocol Buffer inherits a lot from HTTP/2.
|Larger payload over wire||Smaller payload over wire|
|Write your server||.proto files can create server stub|
- As we already know, HTTP/2 is binary. So are protocol buffers. Now you don’t need to pass a date as a string in JSON. You can pass BSON over the wire.
- It is easy on a network as we only use the space we need to use. Say int32 only uses 4 bytes of data. The same data in JSON (string) could have used multiple bytes for a longer integer value.
- We write .proto files. the proto compiler generates stub files which can be used to write the server as well as the client. The key takeaway here is we can use the same proto files to generate clients and servers in many different languages.
- Protocol Buffers are agnostic to the language you are working with to develop your server or the client. .proto files have their syntax and data types which convert to specific data types in a destination programming language. This is my favorite reason to use gRPC in my projects.
I recommend you to read through the proto syntax, keywords, and data types here: https://developers.google.com/protocol-buffers/docs/proto3
What is gRPC?
Ever heard of RPC? It stands for Remote Procedure Call. It’s an old way of running a remote procedure on a remote machine. Let me make it a little familiar for you. When you hit an endpoint from a frontend to a backend, you are making a remote call or a remote procedure call.
SOAP and REST both are an example of RPC. You can send data in the body and hit an API on the other end. That’s how it has been happening since the start.
gRPC is a continuation of that SOAP and REST. gRPC brings all the advancements that their ancestors can’t. With gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object.
Types of Service Methods in gRPC
We’ll practically see what service methods are when we do hands-on with code. But right now I want to specify that in gRPC we have 4 kinds of method definitions.
Unary - Unary is similar to a normal REST call. A client initiates a TCP connection, sends a message, waits for the server to respond, and finally, the server responds.
Server Streaming - Server streaming RPCs where the client sends a request to the server and gets a stream to read a sequence of messages back. For example, you searched for a keyword. Instead of returning a static page, Twitter returns a stream of tweets, including whatever is being tweeted in real time.
Client Streaming - Client streaming RPCs where the client writes a sequence of messages and sends them to the server. Once the client has finished writing the messages, it waits for the server to read them and return its response. Again gRPC guarantees message ordering within an individual RPC call. An example of it would be an IoT device (e.g. a car) streaming its device location to the central server (e.g. Uber).
Bidirectional Streaming - Bidirectional streaming RPCs where both sides send a sequence of messages using a read-write stream. The two streams operate independently, so clients and servers can read and write in whatever order they like. An example of this would be a chat application. But I know there are more complex use cases available in the wild. Please let me know in the comments if you do so.
For a detailed explanation of what happens when a gRPC client calls a gRPC server method, please consider reading RPC life cycle.
Developing with gRPC
We have had enough of theories. Let’s develop some code to see this thing in action.
We are going to have an example in Go. Although as we already know, the proto files have a language of their own which is used to generate code in multiple languages.
We are going to write a simple calculator app.
Write the proto file
.proto files are contracts between the server and the client. This is similar to the REST API you have already experienced with.
Before I proceed further, I’d like to inform you that having the whole code into a module will make your life easy. That’s the reason, I’ve created a go module at the root of the directory named
Protocol buffer data is structured as messages, where each message is a small logical record of information containing a series of name-value pairs called fields. Line 14-17 is an example of a message. So is 20-22. Messages are comprised of data types, identifiers, and index positions.
You’d also note messages are composed inside Services. Service is simply what this web service does. We currently have a
Calculator service from lines 8-11. Calculator service comprises of a method called
Add. Add takes 2 parameters as defined in Input message and emits an Output message.
Generate code from a proto file
You’d need a
protoc compiler to generate code from proto files. If you are on Debian based system, you can use
sudo apt install protobuf-compiler. If you are on any other OS, please read Protocol Buffer Compiler Installation
I am going to use Go for this tutorial, so I’m going to install the Go plugin for the protoc compiler.
go install google.golang.org/protobuf/cmd/[email protected] go install google.golang.org/grpc/cmd/[email protected]
Now with everything in place, I’d issue this command from the root of the module:
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative calculator/calculator.proto
If this command is successful, this would generate 2 more go files.
$ tree calculator calculator ├── calculator_grpc.pb.go ├── calculator.pb.go └── calculator.proto 0 directories, 3 files
If this is your first time with gRPC, and you look at the generated code now, you’d probably be lost.
We are going to use both files to create our server and the client. The generated code will start to make sense.
Implement calculator server and the client
We all know that the output generated by the
protoc compiler is a stub. We need to use that stub as a guide to implementing/writing our client and the server code. The stub work as a guideline for us.
Right now, the generated code lives inside
calculator package in calculator.pb.go which mostly deals with data part (i.e.
Operand2, their getters etc) and calculator_grpc.pb.go which mostly deals with method implementation (i.e
Add in both server and client). The first thing in both server and client code is to import this package.
I will keep referencing the proto file as we define our client/server code.
Write gRPC calculator server
If you look into calculator_grpc.pb.go, you’d find that there is a struct called
UnimplementedCalculatorServer. This struct represents our server. Now there is a reason it is named
Unimplemented. If you look at methods attached to this struct, you’d see a method named
Add. This is the same method we defined in our proto file. Here is a refresher:
// Adds two number rpc Add(Input) returns (Output);
What we are going to do is we are going to take this
UnimplementedCalculatorServer and implement the
Pay attention to the signature of the method. We are taking
Input in form of
*pb.Input and returning
Output in form of
*pb.Output. This is very the same as we declared in the proto file.
Add implementation is incomplete without Add logic. On line 26 we are using
GetOperand2 which is available from the calculator.pb.go file. At last, we use the Output struct to return the result.
Implementing the method is not enough, we also need to start the gRPC server and start listening.
On line 32 I’m starting to listen to TCP connection on a given port.
grpc.NewServer() calls the internal library function to create a new gRPC server, this call returns a
grpc.ServiceRegistrar object. This object is then passed to
RegisterCalculatorServer along with our implemented methods.
The whole code looks like this:
In above implementation, we have used
flag library to pass port from command line.
Write gRPC calculator client
Like we have done with server code, we’ll start by defining command line
Line 19-20 here are kind of a hack as flag module does not have a way to accept
int32 which is requirement for our
Next is connecting with the server part:
grpc.Dial takes address of the server as well as other variadic parameters. As gRPC works over HTTP2 which is by default requires TLS, we are using insecure credentials as we have not configured our server to use certificates.
On the next line, we are going to create a new gRPC client:
On preceding lines, we are going to invoke the
The entirity of the code looks like this:
Let’s go ahead and test our service.
Demo our server and the client
I have recorded a gif for the demo.
Here I have demonstrated with the default flag value, but you can override
-op2 flag to see different results.
This was merely an introduction to gRPC and protocol buffer. What I’ve found is different than a conventional REST API is the endpoints. In REST, we have a predefined endpoint to hit. Such as
/calculation/add. We would then pass JSON with the first and the second operand.
This is not the case with gRPC. We get the stub files as the
protoc artifact. The only communication is from there.
In this post, I have only covered the Unary service method types. I’ll leave streaming for some other time. If you’d like to read and practice more gRPC, I’d found this series very helpful.