Skip to content

Commit 7976ad0

Browse files
peter-smith-phdpeter-smith-phd
committed
Convert gitlab-ci
1 parent 3b5ead3 commit 7976ad0

1 file changed

Lines changed: 101 additions & 59 deletions

File tree

‎src/content/docs/aws/ci-pipelines/gitlab-ci.md‎

Lines changed: 101 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,20 @@ sidebar:
66
order: 6
77
---
88

9-
This page contains easily customisable snippets to show you how to manage LocalStack in a GitLab CI pipeline.
9+
This page contains easily customizable snippets to show you how to manage LocalStack in a GitLab CI pipeline with the [`lstk` CLI](/aws/developer-tools/running-localstack/lstk/).
1010

11-
## Snippets
12-
13-
### Start up Localstack
11+
GitLab runs your job in one container and the Docker daemon in another, so every snippet below pairs the job with a Docker-in-Docker (`dind`) service.
1412

1513
:::tip
1614
While working with a Docker-in-Docker (`dind`) setup, the Docker runner requires `privileged` mode.
1715
You must always use `privileged = true` in your GitLab CI's `config.toml` file while setting up LocalStack in GitLab CI runners.
1816
For more information, see [GitLab CI Docker-in-Docker](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker-executor) documentation.
1917
:::
2018

19+
## Snippets
20+
21+
### Start up LocalStack
22+
2123
LocalStack requires a [CI Auth Token](https://app.localstack.cloud/workspace/auth-tokens), which you must add to the repository's environment variables as `LOCALSTACK_AUTH_TOKEN`.
2224
Go to your project's **Settings > CI/CD** and expand the **Variables** section.
2325
Select the **Add Variable** button and fill in the necessary details with `LOCALSTACK_AUTH_TOKEN` as the key and your CI Auth Token as the value.
@@ -26,141 +28,181 @@ After you create the variable, you can use it in the `.gitlab-ci.yml` file.
2628
However, variables set in the GitLab UI are not automatically passed down to service containers.
2729
You need to assign them as variables in the UI, and then re-assign them in your `.gitlab-ci.yml`.
2830

29-
#### Service
31+
#### Container
32+
33+
In this setup, `lstk` owns the emulator's lifecycle: `DOCKER_HOST` points it at the `dind` daemon, and `lstk start` runs the emulator container there.
3034

3135
```yaml showshowLineNumbers
32-
...
33-
variables:
34-
LOCALSTACK_AUTH_TOKEN: $LOCALSTACK_AUTH_TOKEN
35-
DOCKER_SOCK: tcp://docker:2375
36-
DOCKER_HOST: tcp://docker:2375
37-
DOCKER_TLS_CERTDIR: ""
38-
...
39-
services:
40-
- name: localstack/localstack-pro:latest
41-
alias: localstack
42-
- name: docker:dind
43-
alias: docker
44-
command: ["--tls=false"]
45-
...
36+
image: node:22
37+
38+
stages:
39+
- job
40+
41+
job:
42+
stage: job
43+
variables:
44+
DOCKER_HOST: tcp://docker:2375
45+
DOCKER_TLS_CERTDIR: ""
46+
LOCALSTACK_AUTH_TOKEN: $LOCALSTACK_AUTH_TOKEN
47+
LOCALSTACK_HOST: localhost.localstack.cloud:4566
48+
49+
services:
50+
- name: docker:dind
51+
alias: docker
52+
command: ["--tls=false"]
53+
54+
before_script:
55+
- npm install -g @localstack/lstk
56+
- apt-get update && apt-get install -y awscli
57+
- dind_ip="$(getent hosts docker | cut -d' ' -f1)"
58+
- echo "${dind_ip} localhost.localstack.cloud" >> /etc/hosts
59+
- lstk setup aws
60+
script:
61+
- lstk start
62+
- lstk aws s3 mb s3://test-bucket
63+
- lstk aws s3 ls
4664
```
4765
48-
#### Container
66+
`lstk start` pulls the image, validates your license, and returns only once the emulator is ready, so no separate wait step is needed.
67+
Because the emulator runs on the `dind` daemon, its ports are published on the `docker` service rather than on the job container.
68+
The `/etc/hosts` entry and `LOCALSTACK_HOST` are what let `lstk` and your tests reach it at `localhost.localstack.cloud:4566`; without them `lstk` falls back to `127.0.0.1`, where nothing is listening.
69+
70+
:::note
71+
`lstk` bind-mounts the Docker socket into the emulator, and sets the emulator's own `DOCKER_HOST`, only when it reaches the daemon over a Unix socket.
72+
A TCP `dind` daemon has no socket to mount, so services that spawn their own containers (Lambda, ECS, EKS) need the daemon address passed in explicitly.
73+
`lstk start` forwards `LOCALSTACK_`-prefixed variables to the emulator, which strips the prefix, so set `LOCALSTACK_DOCKER_HOST` to the `dind` daemon as seen from inside the `dind` network (its bridge gateway, usually `tcp://172.17.0.1:2375`).
74+
:::
75+
76+
#### Service
77+
78+
Alternatively, run LocalStack as a GitLab service container and use `lstk` purely as a client, pointing it at the service with `LSTK_ENDPOINT_URL`.
79+
GitLab passes the job's `variables` to service containers too, so the emulator picks up both the auth token and the Docker connection directly, with no prefixing required.
4980

5081
```yaml showshowLineNumbers
51-
image: docker:latest
82+
image: node:22
5283
5384
stages:
5485
- job
5586
5687
job:
5788
stage: job
5889
variables:
59-
...
60-
LOCALSTACK_AUTH_TOKEN: $LOCALSTACK_AUTH_TOKEN
90+
DOCKER_SOCK: tcp://docker:2375
6191
DOCKER_HOST: tcp://docker:2375
6292
DOCKER_TLS_CERTDIR: ""
63-
AWS_ENDPOINT_URL: "http://localhost.localstack.cloud:4566"
64-
...
93+
LOCALSTACK_AUTH_TOKEN: $LOCALSTACK_AUTH_TOKEN
94+
LSTK_ENDPOINT_URL: http://localstack:4566
6595
6696
services:
97+
- name: localstack/localstack-pro:latest
98+
alias: localstack
6799
- name: docker:dind
68100
alias: docker
69101
command: ["--tls=false"]
70102
71103
before_script:
72-
- apk update
73-
- apk add gcc musl-dev linux-headers py3-pip python3 python3-dev
74-
- python3 -m pip install localstack awscli
104+
- npm install -g @localstack/lstk
105+
- apt-get update && apt-get install -y awscli curl
106+
- |
107+
for _ in $(seq 1 60); do
108+
curl -sf "${LSTK_ENDPOINT_URL}/_localstack/health" > /dev/null && break
109+
sleep 2
110+
done
75111
script:
76-
- docker pull localstack/localstack-pro:latest
77-
- dind_ip="$(getent hosts docker | cut -d' ' -f1)"
78-
- echo "${dind_ip} localhost.localstack.cloud " >> /etc/hosts
79-
- DOCKER_HOST="tcp://${dind_ip}:2375" localstack start -d
112+
- lstk aws s3 mb s3://test-bucket
113+
- lstk aws s3 ls
80114
```
81115

82-
You can check the logs of the LocalStack container to see if the activation was successful.
83-
If the CI Auth Token activation fails, LocalStack container will exit with an error code.
116+
GitLab starts service containers before the job's first command, but does not wait for them to become ready, hence the health poll.
84117

85-
### Dump Localstack logs
118+
### Dump LocalStack logs
86119

87120
```yaml showshowLineNumbers
88121
...
89122
job:
90-
variables:
91-
LOCALSTACK_HOST: <LS_HOST>:<LS_PORT>
92123
script:
93-
- localstack logs | tee localstack.log
94-
...
124+
- set +e
125+
- <your test command>; status=$?
126+
- lstk logs --verbose | tee localstack.log
127+
- exit $status
128+
artifacts:
129+
when: always
130+
paths:
131+
- localstack.log
132+
...
95133
```
96134

97-
In case of the service setup `LOCALSTACK_HOST` will be `localstack:4566`.
135+
Collect the logs as the last `script` step rather than in `after_script`, where the emulator container is no longer reachable.
136+
Capturing the test command's exit code keeps the job's result intact while still writing the logs after a failing test, which is when they matter most.
137+
138+
In the [Service](#service) setup, `lstk logs` is not available, because `lstk` does not manage the service container.
139+
Set `CI_DEBUG_SERVICES: "true"` to have GitLab stream the service container's logs into the job log instead.
98140
99-
### Store Localstack state
141+
### Store LocalStack state
100142
101-
You can preserve your AWS infrastructure with Localstack in various ways.
143+
You can preserve your AWS infrastructure with LocalStack in various ways.
102144
103145
#### Artifact
104146
105147
```yaml showshowLineNumbers
106148
...
107149
job:
108150
before_script:
109-
- (test -f ./ls-state-pod.zip && localstack state import ./ls-state-pod.zip) || true
151+
- (test -f ./ls-state.snapshot && lstk load ./ls-state.snapshot --merge=overwrite) || true
110152
script:
111153
...
112-
- localstack state export ./ls-state-pod.zip
154+
- lstk save ./ls-state.snapshot
113155
...
114156
artifacts:
115157
paths:
116-
- $CI_PROJECT_DIR/ls-state-pod.zip
158+
- $CI_PROJECT_DIR/ls-state.snapshot
117159
...
118160
```
119161

120-
More info about Localstack's state export and import [here](/aws/developer-tools/snapshots/saving-snapshots-locally/).
162+
More info about LocalStack's snapshots [here](/aws/developer-tools/snapshots/saving-snapshots-locally/).
121163

122164
#### Cache
123165

124166
```yaml showshowLineNumbers
125167
...
126168
job:
127169
before_script:
128-
- (test -f ./ls-state-pod.zip && localstack state import ./ls-state-pod.zip) || true
170+
- (test -f ./ls-state.snapshot && lstk load ./ls-state.snapshot --merge=overwrite) || true
129171
script:
130172
...
131-
- localstack state export ./ls-state-pod.zip
173+
- lstk save ./ls-state.snapshot
132174
...
133175
cache:
134176
key:
135177
untracked: true
136178
files:
137-
- $CI_PROJECT_DIR/ls-state-pod.zip
179+
- $CI_PROJECT_DIR/ls-state.snapshot
138180
paths:
139-
- $CI_PROJECT_DIR/ls-state-pod.zip
181+
- $CI_PROJECT_DIR/ls-state.snapshot
140182
...
141183
```
142184

143-
Additional information about state export and import [here](/aws/developer-tools/snapshots/saving-snapshots-locally/).
185+
Additional information about snapshots [here](/aws/developer-tools/snapshots/saving-snapshots-locally/).
144186

145187
#### Cloud Pod
146188

147189
```yaml showshowLineNumbers
148190
...
149191
job:
150192
before_script:
151-
- localstack pod load <POD_NAME> || true
193+
- lstk load pod:<POD_NAME> || true
152194
script:
153195
...
154-
- localstack pod save <POD_NAME>
196+
- lstk save pod:<POD_NAME>
155197
...
156198
```
157199

158-
Find more information about cloud pods [here](/aws/developer-tools/snapshots/cloud-pods).
200+
Find more information about Cloud Pods [here](/aws/developer-tools/snapshots/cloud-pods).
159201

160202
## Current Limitations
161203

162-
- Localstack must be able to reach a docker socket to provision containers for certain services, ie Lambda, EKS, ECS...etc
163-
- the runner must be able to resolve the Localstack domain (by default _localhost.localstack.cloud_), see the sample pipelines for a possible solution
164-
- to be able to separate steps into their own jobs one must preserve Localstack's state, since Gitlab is not preserving job related containers/services during the pipelines
165-
- to start up Localstack in Gitlab CI Docker tools are necessary
166-
- when Localstack run as a container, it's not accessible during the `after_script` phase
204+
- LocalStack must be able to reach a Docker socket to provision containers for certain services, such as Lambda, EKS, and ECS.
205+
- The runner must be able to resolve the LocalStack domain (by default _localhost.localstack.cloud_); see the sample pipelines for a possible solution.
206+
- To separate steps into their own jobs, you must preserve LocalStack's state, since GitLab does not preserve job-related containers or services across a pipeline.
207+
- Docker tooling is necessary to start up LocalStack in GitLab CI.
208+
- When LocalStack runs as a container, it is not accessible during the `after_script` phase.

0 commit comments

Comments
 (0)