Go E2E Tutorial Part 4: Twirp RPC

7 min readJul 30, 2023

In this part we will be learning on how to setup and serve REST API using Twirp. The files are hosted here for your reference: https://github.com/hariesef/myschool

Link to previous chapter: https://hariesef.medium.com/go-e2e-tutorial-part-3-unit-testing-with-ginkgo-and-gomega-for-gorm-implementation-sqlite-473b5bb0a625

Previously we have implemented model and repo of Student. Now for learning purposes we want to serve these repo functions in form of REST API. If we look back at our previous clean architecture diagram:

Our Student Repo is meant to be used inside a Service, before being accessed by controller/ REST API. But for this example we are skipping the Service creation (Controller will call interface of Repo directly), to quickly show how easy Twirp is. The Service creation will be explained in other chapter.

About Twirp

Twirp was developed by Twitch to internally replace gRPC. They had been using gRPC previously mainly for machine-to-machine communication but was overwhelmed by complexity and bugs. So what makes Twirp probably be better than gRPC?

Runs on top of HTTP1.1 and HTTP2.0 protocols
Lightweight but powerful
Very quick to setup and use
can serve REST API using Json format

What I like most about Twirp is that, serving REST API using Twirp, we don’t really care about the HTTP layer at all. We don’t have to do HTTP arguments and payload handling at all. Just focus on business logic!

More about twirp: https://twitchtv.github.io/twirp/docs/intro.html

Creating Proto File

In previous chapter we have implemented the functions of Create(), Read(), Delete() and FindByName() in repository. Let us serve this functions over Twirp. First step is to write student.proto file definition. Where to put the file?

The student.proto file is put under pkg/controller/rpc/student/ folder. The other files (.pb and .twirp) are generated by a tool, which will be covered in a minute.

Here is what the student.proto looks like:

syntax = "proto3";

package myschool;
option go_package = "/pkg/controller/rpc/student";

// provides endpoints related with Student information
// this RPC simply to provide example on how to access the repo we have implemented
service Student {
    rpc Create(StudentParam) returns (StudentModel);
    rpc Read(StudentID) returns (StudentModel);
    rpc Delete(StudentID) returns (StudentModel);
    rpc FindByName(StudentName) returns (StudentModels);
}


message StudentID {
    int32 id = 1;
}

message StudentName {
    string name = 1;
}


message StudentParam {
    string name = 1;
    string gender = 2;
}

message StudentModel {
    int32 id = 1;
    int32 created_at = 2;
    int32 updated_at = 3;
    string name = 4;
    string gender = 5;
}

message StudentModels {
    repeated StudentModel students = 1;
}

Explanation:

package “myschool” defines the application/ top level name. This name will show on REST API path.
go_package denotes the package path, where the go package folder resides.
service defines the service name, and what are the methods available.
For each argument and return value, they have to be defined as struct.
Special for FindByName, because it return arrays, in the definition of StudentModels, we have to specify “repeated <type> <field>”. This actually means, field students is array of StudentModel.

Next step is to generate the pb and twirp files, which can be done via this command:

 protoc --twirp_out=. --go_out=. pkg/controller/rpc/student/student.proto

PS: The command is also available in Makefile: https://github.com/hariesef/myschool/blob/master/Makefile

student.twirp.go contains generated implementation of Server and Client, for both protobuf and Json protocol. We mainly need this file to instantiate a handler to be served by Go default http server. It also contains interface of functions (Create(StudentParam), Read(StudentID), etc) as defined in proto file, that we have to implement very soon.
student.pb.go contains generated struct implementation (StudentParam, StudentID, StudentName, etc)

Intermezzo: Repositories

Previously we already have the Student Repo. That is one repo, but we will have more repositories implementation, nonetheless. All repos can be collected into one struct Repositories.

The file can be inspected at https://github.com/hariesef/myschool/blob/master/internal/repositories/repositories.go

type Repositories struct {
 .......
 StudentRepo   model.StudentRepo
 UserRepo      model.UserRepo
 AuthTokenRepo model.AuthTokenRepo
 .......
}

The Repositories object can be constructed directly like, when we only need studentRepo for unit test:

  studentRepoImpl := sqLiteStudent.NewRepo(db)
  repo = &repositories.Repositories{
   StudentRepo: studentRepoImpl,
  }

But in real execution, the repositories package will provide Setup() function to be called by main.go, where it will return with actual connection to database. Aside from this, repositories folder does not contain any other file or folders.

The repositories is important object to explain here, because it is one of dependency for Twirp implementation and Service implementation.

Implement Twirp Methods

We will to create files under this folder:

Here is how the Create() method is implemented under student_rpc_impl.go:

import (
 "context"

 pb "myschool/pkg/controller/rpc/student"

 "myschool/internal/repositories"
 "myschool/pkg/model"

 "github.com/twitchtv/twirp"
)

type StudentRPCServer struct {
 Repo *repositories.Repositories
}

func (s *StudentRPCServer) Create(ctx context.Context, sp *pb.StudentParam) (*pb.StudentModel, error) {

 studentParam := model.StudentCreationParam{
  Name:   sp.Name,
  Gender: sp.Gender,
 }
 studentModel, err := s.Repo.StudentRepo.Create(ctx, studentParam)
 if err != nil {
  return nil, twirp.InternalError(err.Error())
 }

 return &pb.StudentModel{
  Id:        int32(studentModel.GetUID()),
  CreatedAt: int32(studentModel.GetCreatedAt()),
  UpdatedAt: int32(studentModel.GetUpdatedAt()),
  Name:      studentModel.GetName(),
  Gender:    studentModel.GetGender(),
 }, nil
}

If variable type starts with pb, means it is struct from file student.pb.go.
To construct the StudentRPCServer (implementation of student.twirp.proto), this file has to implement all required methods: Create(), Read(), Delete() and FindByName().
To construct the StudentRPCServer it requires Repo, instance of Repositories we discussed above. The Repo will give access to actual function to create student in database.
The StudentRPCServer construction using Repo is just an example. Normally what is required to construct, is interface of Service, not a Repo. Service construction itself, then will require Repo.
All methods declared here are basically serving as proxy to actual functions. I.e. Twirp.Create() is calling Repo.Create(). In next tutorial, the Twirp.Function() will call Service.Function() instead.
Because only serving as proxy, there should be NO BUSINESS LOGIC at all implemented here. Pure just call service/repo functions and that’s it.
For return value, because the type is pb.StudentModel, then we have to transform the values out of original studentModel.

The rest of methods implementation can be seen here: https://github.com/hariesef/myschool/blob/master/internal/controller/rpc/student/student_rpc_impl.go

Serving the Twirp as API

When we already have Twirp service methods implemented, it is now time to bring them up, served as HTTP endpoints. Setting up twirp and http servers are done in main logic, cmd.go:

Please refer to this file for full content: https://github.com/hariesef/myschool/blob/master/cmd/server/main.go

First we have to import the required packages

import (
 .....
 
 studentRPCImpl "myschool/internal/controller/rpc/student"
 studentRPCIface "myschool/pkg/controller/rpc/student"
 "net/http"
 "github.com/dewanggasurya/logger/log"
 
 .....
)

studentRPCIface will refer to our student.twirp.go file.
studentRPCImpl will refer to our implementation above.

Next is to instantiate server and handler:

 studentServer := &studentRPCImpl.StudentRPCServer{Repo: repo}
 studentTwirpHandler := studentRPCIface.NewStudentServer(studentServer,
  rpc.TwirpHookOption(studentRPCIface.StudentPathPrefix))
 studentTwirpHandler2 := rpc.WithEvaluateHeaders(studentTwirpHandler)

studentServer is instance of our implementation.
studentTwirpHandler needs studentServer as the implementation.
NewStudentServer() is method generated from the proto file.
studentTwirpHandler2 is new handler, after we added two more optional features: Hook and Headers, which will be explained shortly later.

Because we will host more than one Twirp services, we need an HTTP Muxer/ Router. We will be using Gochi. Then the rest is just spinning up Go default http.Server{} and place the Gochi router as the handler.

 router := chi.NewRouter()
 router.Mount(studentRPCIface.StudentPathPrefix, studentTwirpHandler2)

 server := &http.Server{
  Addr:    fmt.Sprintf("0.0.0.0:%d", 8080),
  Handler: router,
 }

 go func(srv *http.Server) {
  log.Infof("listen on port : %d", 8080)
  if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
   log.Infof("listen: %s", err.Error())
  }
 }(server)

Optional Twirp Hooks

In above example, we are using rpc.TwirpHookOption() as parameter in setting up Twirp handler. This is actually function that allows us to customize what to do, when we

receive a request
error happens during processing the request
successfully sent a response

In our case, we are demonstrating the hooks to populate some debug logs. Complete file is implemented under pkg/rpc/common.go. Please refer here for full content: https://github.com/hariesef/myschool/blob/master/pkg/controller/rpc/common.go

func TwirpHookOption(path string) twirp.ServerOption {
 path = path[1 : len(path)-1]
 return twirp.WithServerHooks(&twirp.ServerHooks{
  RequestRouted: func(ctx context.Context) (context.Context, error) {
   method, _ := twirp.MethodName(ctx)
   log.Debugf("[twirp][%s] incoming method : %v", path, method)
   return ctx, nil
  },
  Error: func(ctx context.Context, twerr twirp.Error) context.Context {
   log.Debugf("[twirp][%s] error (%v) : \"%v\"", path, string(twerr.Code()), twerr.Error())
   return ctx
  },
  ResponseSent: func(ctx context.Context) {
   log.Debugf("[twirp][%s] response sent", path)
  },
 })
}

Optional Twirp Headers Processing

Checking the headers of incoming request is inevitable in modern system. We may need to check headers for user-agent, IP address, Token, client ID, client secret, etc.

Just like hooks handling, Twirp optionally provides this functionality for us to customize:

func WithEvaluateHeaders(base http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  ctx := r.Context()
  ua := r.Header.Get("User-Agent")
  token := r.Header.Get("Token")
  ctx = context.WithValue(ctx, "user-agent", ua)
  ctx = context.WithValue(ctx, "token", token)
  r = r.WithContext(ctx)

  base.ServeHTTP(w, r)
 })
}

The function above is still part of the pkg/controller/rpc/common.go

In that custom function, we are getting user-agent and token from request headers, and store them inside the context. This context is then transferred to our Twirp service methods, which also will be redirected to Service functions. Then under the Service implementation where business logic resides, we can read the headers from context and evaluate them as needed.

Please refer to this file, scroll down to Logout() function, there is an example on how to get the header. The http request header is also available on postman test case file Logging out an account.

Let’s test the Twirp with Postman!

Once we run the application locally with go run cmd/server/main.go we will be able to use postman to send request to localhost port 8080. The script is also available in github: https://github.com/hariesef/myschool/blob/master/MySchool.postman_collection.json

You can import the script into your own postman account and start testing.