Android Overlayfs integration with adb remount

Introduction

Users working with userdebug or eng builds expect to be able to
remount the system partition as read-write and then add or modify
any number of files without reflashing the system image, which is
understandably efficient for a development cycle.
Limited memory systems that chose to use readonly filesystems like
squashfs, or Logical Resizable Android Partitions which land
system partition images right-sized, and with filesystem that have
been deduped on the block level to compress the content; means that
either a remount is not possible directly, or when done offers
little or no utility because of remaining space limitations or
support logistics.

Overlayfs comes to the rescue for these debug scenarios, and logic
will automatically setup backing storage for a writable filesystem
as an upper reference, and mount overtop the lower. These actions
will be performed in the adb disable-verity and adb remount
requests.

Operations

Cookbook

The typical action to utilize the remount facility is:

$ adb root
$ adb disable-verity
$ adb reboot
$ adb wait-for-device
$ adb root
$ adb remount

Followed by one of the following:

$ adb stop
$ adb sync
$ adb start
$ adb reboot

or

$ adb push <source> <destination>
$ adb reboot

Note that the sequence above:

$ adb disable-verity
$ adb reboot

or

$ adb remount

can be replaced in both places with:

$ adb remount -R

which will not reboot if everything is already prepared and ready
to go.

None of this changes if overlayfs needs to be engaged.
The decisions whether to use traditional direct filesystem remount,
or one wrapped by overlayfs is automatically determined based on
a probe of the filesystem types and space remaining.

Backing Storage

When overlayfs logic is feasible, it will use either the
/cache/overlay/ directory for non-A/B devices, or the
/mnt/scratch/overlay directory for A/B devices that have
access to Logical Resizable Android Partitions.
The backing store is used as soon as possible in the boot
process and can occur at first stage init, or at the
mount_all init rc commands.

This early as possible attachment of overlayfs means that
sepolicy or init itself can also be pushed and used after
the exec phases that accompany each stage.

Caveats

• Space used in the backing storage is on a file by file basis
  and will require more space than if updated in place. As such
  it is important to be mindful of any wasted space, for instance
  BOARD_IMAGE_PARTITION_RESERVED_SIZE being defined
  will have a negative impact on the overall right-sizing of images
  and thus free dynamic partition space.
• Kernel must have CONFIG_OVERLAY_FS=y and will need to be patched
  with "*overlayfs: override_creds=off option bypass creator_cred*"
  if kernel is 4.4 or higher.
  The patch is available on the upstream mailing list and the latest as of
  Feb 8 2019 is https://lore.kernel.org/patchwork/patch/1009299/.
  This patch adds an override_creds mount option to overlayfs that
  permits legacy behavior for systems that do not have overlapping
  sepolicy rules, principals of least privilege, which is how Android behaves.
• adb enable-verity will free up overlayfs and as a bonus the
  device will be reverted pristine to before any content was updated.
  Update engine does not take advantage of this, will perform a full OTA.
• Update engine may not run if fs_mgr_overlayfs_is_setup() reports
  true as adb remount overrides are incompatible with an OTA resources.
• For implementation simplicity on retrofit dynamic partition devices,
  take the whole alternate super (eg: if "*a*" slot, then the whole of
  "*system_b*").
  Since landing a filesystem on the alternate super physical device
  without differentiating if it is setup to support logical or physical,
  the alternate slot metadata and previous content will be lost.
• If dynamic partitions runs out of space, resizing a logical
  partition larger may fail because of the scratch partition.
  If this happens, either fastboot flashall or adb enable-verity can
  be used to clear scratch storage to permit the flash.
  Then reinstate the overrides and continue.
• File bugs or submit fixes for review.