gRPC (Core) Compression Cookbook

Introduction

This document describes compression as implemented by the gRPC C core. See the
full compression specification for details.

Intended Audience

Wrapped languages developers, for the purposes of supporting compression by
interacting with the C core.

Criteria for GA readiness

Be able to set compression at channel,
call and message level.
In principle this API should be based on compression levels as opposed to
algorithms. See the discussion below.
Have unit tests covering the cases from the
spec.
Interop tests implemented and passing on Jenkins. The two relevant interop
test cases are
large_compressed_unary
and
server_compressed_streaming.

Summary Flowcharts

The following flowcharts depict the evolution of a message, both incoming and
outgoing, irrespective of the client/server character of the call. Aspects
still not symmetric between clients and servers (e.g. the use of compression
levels)
are explicitly marked. The in-detail textual description for the different
scenarios is described in subsequent sections.

Incoming Messages

Outgoing Messages

Levels vs Algorithms

As mentioned in the relevant discussion on the spec
document,
compression levels are the primary mechanism for compression selection at the
server side. In the future, it'll also be at the client side. The use of levels
abstracts away the intricacies of selecting a concrete algorithm supported by a
peer, on top of removing the burden of choice from the developer.
As of this writing (Q2 2016), clients can only specify compression algorithms.
Clients will support levels as soon as an automatic retry/negotiation mechanism
is in place.

Per Channel Settings

Compression may be configured at channel creation. This is a convenience to
avoid having to repeatedly configure compression for every call. Note that any
compression setting on individual calls or
messages overrides channel settings.

The following aspects can be configured at channel-creation time via channel arguments:

Disable Compression Algorithms

Use the channel argument key
GRPC_COMPRESSION_CHANNEL_ENABLED_ALGORITHMS_BITSET (from
grpc/impl/codegen/compression_types.h),
takes a 32 bit bitset value. A set bit means the algorithm with that enum value
according to grpc_compression_algorithm is enabled.
For example, GRPC_COMPRESS_GZIP currently has a numeric value of 2. To
enable/disable GZIP for a channel, one would set/clear the 3rd LSB (eg, 0b100 =
0x4). Note that setting/clearing 0th position, that corresponding to
GRPC_COMPRESS_NONE, has no effect, as no-compression (a.k.a. identity) is
always supported.
Incoming messages compressed (ie, encoded) with a disabled algorithm will result
in the call being closed with GRPC_STATUS_UNIMPLEMENTED.

Default Compression Level

(currently, Q2 2016, only applicable for server side channels. It's ignored
for clients.)
Use the channel argument key GRPC_COMPRESSION_CHANNEL_DEFAULT_LEVEL (from
grpc/impl/codegen/compression_types.h),
valued by an integer corresponding to a value from the grpc_compression_level
enum.

Default Compression Algorithm

Use the channel argument key GRPC_COMPRESSION_CHANNEL_DEFAULT_ALGORITHM (from
grpc/impl/codegen/compression_types.h),
valued by an integer corresponding to a value from the grpc_compression_level
enum.

Per Call Settings

Compression Level in Call Responses

The server requests a compression level via initial metadata. The
send_initial_metadata grpc_op contains a maybe_compression_level field
with two fields, is_set and compression_level. The former must be set when
actively choosing a level to disambiguate the default value of zero (no
compression) from the proactive selection of no compression.

The core will receive the request for the compression level and automatically
choose a compression algorithm based on its knowledge about the peer
(communicated by the client via the grpc-accept-encoding header. Note that the
absence of this header means no compression is supported by the client/peer).

Compression Algorithm in Call Responses

Server should avoid setting the compression algorithm directly. Prefer
setting compression levels unless there's a very compelling reason to choose
specific algorithms (benchmarking, testing).

Selection of concrete compression algorithms is performed by adding a
(GRPC_COMPRESS_REQUEST_ALGORITHM_KEY, <algorithm-name>) key-value pair to the
initial metadata, where GRPC_COMPRESS_REQUEST_ALGORITHM_KEY is defined in
grpc/impl/codegen/compression_types.h),
and <algorithm-name> is the human readable name of the algorithm as given in
the HTTP2 spec
for Message-Encoding (e.g. gzip, identity, etc.). See
grpc_compression_algorithm_name
for the mapping between the grpc_compression_algorithm enum values and their
textual representation.

Per Message Settings

To disable compression for a specific message, the flags field of grpc_op
instances of type GRPC_OP_SEND_MESSAGE must have its GRPC_WRITE_NO_COMPRESS
bit set. Refer to
grpc/impl/codegen/compression_types.h),