case studies

The Hugo Deploy Command Brought In 18 New Go Modules Including the AWS SDK and gRPC

Hugo, the static site generator written in Go, has a hugo deploy command that uploads a built site to Amazon S3, Google Cloud Storage or Azure Blob Storage. It arrived in commit c7165589 of gohugoio/hugo, “Add a “deploy” command”, written in December 2018 and committed to the main branch in May 2019. It works through gocloud.dev, the Go Cloud Development Kit, which gives one interface over the three storage services.

That’s the right tool for a command that talks to three clouds. It also means the code of all three clouds’ Go libraries comes along. In that commit go.sum went from 178 lines to 744, and the module list Remixware reads from go.mod and go.sum went from 81 modules to 105. We ran Remixware 0.3.1’s dependency gate on the commit against its parent, offline. The commands, hashes and output are in the Remixware repository, and the numbers here come from there.

Two Lines in go.mod, 18 New Modules

remixware deps . --base-ref ad5703a91712cd245231ba8fdbc49632c794a165

go.mod gained two direct requirements, gocloud.dev v0.13.0 and github.com/dustin/go-humanize v1.0.0. The gate counted 18 modules that no lockfile on the base side named, and with no allow list every one of them fails:

NOT ALLOWED          golang/gocloud.dev v0.13.0, added (direct)
                     in go.mod
                     New, and not on the allow list. Add a deps allow rule with a reason to the
                     policy file, or drop it.

The 16 transitive ones are the clouds’ own code and what it needs:

  • Amazon: github.com/aws/aws-sdk-go v1.16.23 and github.com/jmespath/go-jmespath
  • Microsoft: github.com/Azure/azure-storage-blob-go and github.com/Azure/azure-pipeline-go
  • Google: cloud.google.com/go, google.golang.org/api, google.golang.org/grpc, google.golang.org/genproto, both majors of github.com/googleapis/gax-go, github.com/golang/protobuf, github.com/google/uuid and github.com/google/wire
  • Tracing and the Go project: go.opencensus.io, golang.org/x/oauth2 and golang.org/x/xerrors

Six modules Hugo already had moved to new versions, among them golang.org/x/net, golang.org/x/sys and github.com/BurntSushi/toml. Those are updated, which doesn’t fail. The run ended with:

24 changes: 18 not-allowed, 6 updated.
FAIL: 18 changes break the policy (fail on: not-allowed, install-script, publisher-changed, not-checked).

Why Go Gets Only the Allow List

The checks block said the other two checks don’t apply:

  install scripts    not applicable, 24 added or updated versions: 24 not applicable
                     (Go modules run no install scripts)
  publisher changes  not applicable, 24 added or updated versions: 24 not applicable
                     (the Go module proxy records no publisher)

Downloading a Go module runs none of its code, and the Go module proxy keeps no record of who published a version. For a Go change the useful question is the plain one: which code is now part of the build that wasn’t before, and who looked at it. Here the answer covers three cloud SDKs, and a reviewer would want to know that before merging, whatever they decide.

Allowing What You Already Trust

Most teams already trust some publishers. With an example policy that allows the Go project’s own modules and Google’s (written for this run; Hugo doesn’t use Remixware):

{
  "deps": {
    "allow": [
      {"package": "golang/golang.org/x/*", "reason": "modules of the Go project"},
      {"package": "golang/google.golang.org/*", "reason": "Google's Go API, gRPC and protobuf modules"},
      {"package": "golang/cloud.google.com/*", "reason": "Google Cloud client libraries"}
    ]
  }
}

the same change has 12 failing modules instead of 18. What’s left is the part a person should read: gocloud.dev and go-humanize, which the commit asked for by name, plus the Amazon and Azure SDKs, the tracing library and the smaller helpers. Each one that’s accepted gets a rule with a reason, and the next change that touches them starts from that record.

The use examples show this policy at work in a monorepo next to an npm and a pip lockfile, and scripts/deps_case_studies.sh in the repository runs this comparison again from scratch.