Hack
Contributing¶
Welcome to Hacking the Planet with abra
! We're looking forward to see what
you come up. If you have any questions, don't hesitate to ask 💖 If any of your
changes seems a bit controversial, it's probably to come have a chat first to
avoid heartache.
In general, we're into the idea of "Optimistic Merging" (instead of "Pessimistic Merging" based on our understanding of C4 (described further down under "Development Process" and also in this blog post).
Quick start¶
Get a fresh copy of the abra
source code from here.
Install direnv, run cp .envrc.sample .envrc
, then run direnv allow
in this directory. This will set coopcloud repos as private due to this bug.. Or you can run go env -w GOPRIVATE=coopcloud.tech
but I'm not sure how persistent this is.
Install Go >= 1.16 and then:
make build
to build. If this fails, rungo mod tidy
../abra
to run commandsmake test
will run testsmake install-abra
will install abra to$GOPATH/bin
make install-kadabra
will install kadabra to$GOPATH/bin
go get <package>
andgo mod tidy
to add a new dependency
Our Drone CI configuration runs a number of checks on each pushed commit. See the Makefile for more handy targets.
Please use the conventional commit format for your commits so we can automate our change log.
Unit tests¶
Run tests¶
Run the entire suite.
make test
Filter tests¶
Run a specific test.
go test ./pkg/recipe -v -run TestGetVersionLabelLocalDoesNotUseTimeoutLabel
Integration tests¶
Running on the CI server¶
Based on R020, we have automated running the integration test suite. Here's the TLDR;
- We have a donated CI server (tysm
@mirsal
💝) standing at the ready,int.coopcloud.tech
. - We run the entire integration suite nightly via our Drone CI/CD configuration here (see "
name: integration test
" stanza) - Here is the script that is run on the remote server:
run-ci-int
What follows is a listing of how this was achieved so that we can collectivise the maintenance.
On the server, we have:
- Created an
abra
user withdocker
permissions - Ran
apt install bats bats-file bats-assert bats-support jq make git golang-1.21 wget bash
- Installed
bats-core
from source, following the instructions below - Docker was already installed on the machine, so nothing to do there
docker login
with thethecoopcloud
details so we don't get rate limited
The drone configuration was wired up as follows:
- Generated a SSH key and put the public key part in
~/.ssh/authorize_keys
- Added that public key part as a "deploy key" in the abra repo (so we can do
ssh://
git remote pulls) - Added the private key part as a Drone secret which is available in build so that the build can SSH over to the server to run commands. That was done like so:
drone secret add --repository coop-cloud/abra --name abra_int_private_key --data @id_ed25519
- In order to specify a cron timing, you need to create it with the Drone CLI:
drone cron add "coop-cloud/abra" "integration" @daily --branch main
Please ask @decentral1se
or on the Matrix channels for SSH access to the machine.
Running them locally¶
Install dependencies¶
We use bats
to run the tests.
You can install the required dependencies with the following. You also need a
working installation of Docker and Go >= 1.16 (not covered in this section).
apt install bats-file bats-assert bats-support jq make git
Unfortunately, the latest bats
version in Debian stable does not have the
"filter tests by tags" feature, which is very handy for running a subset of the
tests. For this, we need to install bats
from source. It's easy.
apt purge -y bats
git clone https://github.com/bats-core/bats-core.git
cd bats-core
sudo ./install.sh /usr/local
Setup Test Server¶
For some tests an actual server is needed, where apps can be deployed. You can either use a local one or a remote test server. There is also a way to run or skip tests that require a remote server. This is covered below in the filtering tests section.
Remote swarm¶
export ABRA_TEST_DOMAIN="test.example.com"
export ABRA_DIR="$HOME/.abra_test"
ABRA_TEST_DOMAIN
should also have a DNS A record for *.test.example.com
which points to the same server so that the test suite can deploy apps freely.
The test suite does not deploy Traefik for you.
Local swarm¶
When running the test suite localy you need a running docker swarm setup:
docker swarm init
docker network create -d overlay proxy
To use the local swarm set the foloowing env var:
export TEST_SERVER=default
export ABRA_DIR="$HOME/.abra_test"
Run tests¶
Now you can run the whole test suite:
bats -Tp tests/integration
Or you can run a single test file:
bats -Tp tests/integration/autocomplete.bats
Tagging tests¶
When a test actually deploys something, we tag it as "slow". When the test requires public DNS, we use "dns". There may be more tags we write more tests.
# bats test_tags=slow,dns
@test "..." {
...
}
Then we can use filters (see below) to pick out a subset of
tests which do/do not use a live server. Feel free to come up with your own
tags. See the bats-core
docs
for more.
Filter tests¶
You can run a specific file.
bats -Tp tests/integration/autocomplete.bats
For example, if you want to check that all abra recipe ...
tests remain working.
bats -Tp tests/integration/recipe_*
You can filter on test names to run specific kinds of tests.
bats -Tp tests/integration --filter "validate app argument"
You can filter on tags.
bats -Tp tests/integration --filter-tags \!slow # only fast tests
bats -Tp tests/integration --filter-tags slow # only slow tests
bats -Tp tests/integration --filter-tags slow,\!dns # slow but no DNS tests
You can also only run the previously failed tests.
mkdir -p tests/integration/.bats/run-logs
bats -Tp tests/integration # run tests
bats -Tp tests/integration --filter-status failed # re-run only failed
Debug tests¶
If you're running into issues and want to debug stuff, you can pass -x
to
bats
to trace all commands run in the test. You can add echo '...' >&3
debug statements to your test to output stuff also.
Using the abra
public API¶
Warning, there is currently no stability promise for the abra
public API! Most of the internals are exposed in order to allow a free hand for developers to try build stuff. If people start to build things then we can start the discussion on what is useful to have open/closed and keep stable etc. Please let us know if you depend on the APIs!
The pkg.go.dev
documentation is here. Here's a brief example to get you going:
package main
import (
"context"
"fmt"
"log"
abraClient "coopcloud.tech/abra/pkg/client"
dockerClient "github.com/docker/docker/client"
)
func getClient(serverName string) (*dockerClient.Client, error) {
cl, err := abraClient.New(serverName)
if err != nil {
return nil, fmt.Errorf("getClient: %s", err)
}
return cl, nil
}
func main() {
cl, err := getClient("foo.example.com")
if err != nil {
log.Fatal(err)
}
// do stuff with the client...
// https://pkg.go.dev/github.com/docker/docker/client
}
Some tools that are making use of the API so far are:
Cross-compiling¶
If there's no official release for the architecture you use, you can cross-compile abra
very easily. Clone the source code from here and then:
- enter the
abra
directory - run
git tag -l
to see the list of tags, choose the latest one - run
git checkout <tag>
, where<tag>
is the latest version - run
GOOS=<os> GOARCH=<arch> [GOARM=<arm>] make build
. You only have to useGOARM
if you're building for ARM, this specifies the ARM version (5,6,7 etc). See this for a list of all supported OS'es and architectures.
Building in Docker¶
If you are living under a curse of constant Go environment problems, it might be easier to build abra
using Docker:
sudo setenforce 0 # SELinux probably won't allow Docker to access files
docker run -it -v $PWD:/abra golang:1.19.6 bash
cd /abra
. .envrc
git config --global --add safe.directory /abra # work around funky file permissions
make build
Release management¶
We use goreleaser to help us automate releases. We use semver for versioning all releases of the tool. While we are still in the public beta release phase, we will maintain a 0.y.z-beta
format. Change logs are generated from our commit logs. We are still working this out and aim to refine our release praxis as we go.
For developers, while using this -beta
format, the y
part is the "major" version part. So, if you make breaking changes, you increment that and not the x
part. So, if you're on 0.1.0-beta
, then you'd go to 0.1.1-beta
for a backwards compatible change and 0.2.0-beta
for a backwards incompatible change.
Making a new release¶
- Run the integration test suite and the unit tests (
make test
) (takes a while!) - Change
ABRA_VERSION
inscripts/installer/installer
to match the new tag (use semver) - Commit that change (e.g.
git commit -m 'chore: publish next tag x.y.z-beta'
) - Make a new tag (e.g.
git tag -a x.y.z-beta
) - Push the new tag (e.g.
git push && git push --tags
) - Wait until the build finishes on build.coopcloud.tech
- Deploy the new installer script (e.g.
cd ./scripts/installer && make
) - Check the release worked, (e.g.
abra upgrade; abra -v
)
Fork maintenance¶
godotenv
¶
We maintain a fork of godotenv because we need inline comment parsing for environment files. You can upgrade the version here by running go get git.coopcloud.tech/coop-cloud/godotenv@0<COMMID>
where <commit>
is the latest commit you want to pin to. See abra#391
for more.
docker/client
¶
A number of modules in pkg/upstream are copy/pasta'd from the upstream docker/docker/client. We had to do this because upstream are not exposing their API as public.
github.com/schultz-is/passgen
¶
Due to coop-cloud/organising#358
.