The Deploy step has not completed the deployment after downloading the logs for several days. The root cause has nothing to do with GitHub? Changing the CI platform and installing panels cannot save it. This article records the real pitfalls of Actions + Alibaba Cloud ECS full-link CI/CD: from the mirror pulling hell (including the blind spots of the mechanism that registry-mirrors cannot manage ghcr, and the practical operation of Nanda ghcr proxy station) to the SSH long connection keep-alive, port preemption and health check 301 endless loop, all ten pits provide “symptoms → root cause → prescription”, the configuration can be copied, and you will stay up a few nights after reading it.
Write an article about a pitfall I just stepped into this month: using GitHub Actions to automatically deploy my Node + Docker project to Alibaba Cloud ECS. The logic itself isn’t any difficult – build, rsync, SSH up
docker compose up -dtwenty lines of YAML. What’s really annoying isDomestic server mirroring, cross-border SSH long connection, port preemptionThis place is fragmented. I wrote down the complete process and the troubleshooting process for each pitfall – the real situation is that this set of 20 lines of YAML took me back and forth for several days. I hope that the students who read this article can finish it in one day.The reading time is about 12 minutes, and it is all content that has been reproduced in actual combat. The configuration can be copied directly (just replace the angle brackets placeholders with your own).
1. Show the results first: the final running form
git push (main)
│
▼
GitHub Actions Runner(海外)
├─ ① 构建前端 dist(node 22)
├─ ② rsync 同步 backend/ deploy/ dist/ → 服务器 /opt/app/
└─ ③ SSH 执行 deploy/deploy.sh
│
▼ 服务器(阿里云 ECS,Alibaba Cloud Linux)
1/4 flock 互斥锁
2/4 docker compose pull(失败重试 3 次)
3/4 docker compose up -d --build
4/4 60 秒内 /api/health 检查,不健康就报红退出
Daily deployment 2~3 minutesSees green; the old container continues to serve when it fails, and there is no sense online.
The assembly line seemed harmless to humans and animals, but I actually experienced these accidents when I landed:
| Phenomenon | Positioning process | root cause |
|---|---|---|
Deploy steps are repeated for several days Downloading Retry on failure |
Looking at the log, the server is connecting registry-1.docker.io |
Image pull link problem has nothing to do with the CI platform |
exit code 12 |
rsync protocol error | The server does not have rsync installed. |
exit code 18 |
Error during transfer phase | Same as above, mirroring timeout causes chain failure |
| Even with the ghcr accelerator, it still can’t move. | docker info Is the mirror in effect? |
registry-mirrors only takes effect on Docker Hub |
Setup SSH steps bizarre red |
In the log echo "" > deploy_key |
Secrets were accidentally deleted and the template was rendered into an empty string. |
Suddenly when the build is advancing normally exit code 255 |
client_loop: send disconnect: Broken pipe |
Cross-border NAT cuts off ssh long connections |
address already in use :80 |
ss -tlnp see naked nginx process |
dnf security update successfully installed host nginx |
| The container is up but the health check is all red | curl 127.0.0.1 Return 301 |
nginx if ($host != 域名) Accidentally damaged IP direct access, 301 endless loop |
Below, the entire set of plans and corresponding pitfalls will be explained in order of launch.
2. One-time initialization on the server side
2.1 Deploy a dedicated account (don’t let CI run with root)
# 创建部署用户并加入 docker 组(免 sudo)
sudo useradd -m -s /bin/bash deployer
sudo usermod -aG docker deployer
sudo mkdir -p /opt/app && sudo chown deployer:deployer /opt/app
# 授权 CI 公钥(私钥留在自己电脑上,公钥贴进 GitHub Secrets 的是配对的私钥)
sudo -u deployer mkdir -p /home/deployer/.ssh && sudo -u deployer chmod 700 /home/deployer/.ssh
echo "ssh-ed25519 AAAA... ci-deploy-key" | sudo -u deployer tee -a /home/deployer/.ssh/authorized_keys
sudo -u deployer chmod 600 /home/deployer/.ssh/authorized_keys
💡 Trap 0 (many people fall for it in the first step):
sudo -u deployer mkdir ~/.sshinside~is made ofThe current root shell is expanded firstYes, it will actually be created/root/.ssh,directPermission denied. Please use absolute paths when creating directories for other users./home/deployer/.ssh。
Generate key pair locally:
ssh-keygen -t ed25519 -f $HOME\.ssh\app-deploy -N '""'
# 公钥贴服务器 authorized_keys,私钥整段贴进 GitHub Secret DEPLOY_KEY
2.2 Install rsync (synchronization dependency of Actions)
sudo dnf install -y rsync
If you don’t install it, you will get a very stingy error:
rsync: [sender] ... protocol error: exit code 12
rsync exit 12 = There is no rsync command on the remote end. The protocol error does not tell you “the other party does not exist”, but only throws out a number. The first time you see it, you will really doubt your life.
2.3 Docker accelerator: first find out how much scope it manages
/etc/docker/daemon.json:
{
"registry-mirrors": [
"
"
]
}
sudo systemctl daemon-reload && sudo systemctl restart docker
docker info | grep -A 3 "Registry Mirrors" # 必须看到这两行才算生效
Three details:
- The “Image Accelerator” in the Alibaba Cloud Container Image Service console requires a dedicated address.(
xxxxxx.mirror.aliyuncs.com), most of the public addresses copied online are no longer valid; - During the investigation, I saw someone put
This typeMirror included websiteFill in registry-mirrors – it is a directory site for you to search for mirrors, not a registry endpoint. If it is filled in, it will be an invalid entry; - Must be completed after modification
restart docker,docker infoverify,Don’t rely on “the file is written = it takes effect”。
Properly configured accelerators can pull Docker Hub images (nginx, redis, postgres, node) in seconds.But note that it is completely ineffective on the ghcr.io image that is most likely to get stuck in your project.——This is the biggest pitfall of this article, which will be discussed in detail in Section 5.
3. GitHub Actions workflow
storehouse Settings → Secrets and variables → Actions Create 4 Secrets:DEPLOY_HOST(Server IP),DEPLOY_USER(deployer)、DEPLOY_KEY(The entire text of the private key),DEPLOY_PORT(22)。
.github/workflows/deploy.yml(Production Verified Version):
name: deploy-production
on:
push:
branches: [main]
# 纯文档改动不触发部署,省 runner 也省一次重启
paths-ignore:
- '**.md'
- 'doc/**'
# 同一时间只允许一条部署流水线,排队不取消
concurrency:
group: deploy-production
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Build frontend
run: |
cd frontend
npm ci
npm run build
# ubuntu-latest 自带 rsync/openssh,不需要 apt 装(见下方坑 5)
- name: Setup SSH
env:
DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
if [ -z "$DEPLOY_KEY" ] || [ -z "$DEPLOY_HOST" ]; then
echo "::error::Secrets DEPLOY_KEY/DEPLOY_HOST 未配置"
exit 1
fi
mkdir -p ~/.ssh
printf '%s\n' "$DEPLOY_KEY" > ~/.ssh/deploy_key
chmod 600 ~/.ssh/deploy_key
ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts 2>/dev/null
- name: Sync to server
run: |
rsync -az --delete \
--exclude '.git' \
--exclude 'node_modules' \
--exclude '.env' \
--exclude 'backend/uploads' \
--exclude 'backend/data' \
-e "ssh -i ~/.ssh/deploy_key -p ${DEPLOY_PORT:-22} -o ServerAliveInterval=30 -o ServerAliveCountMax=20" \
./ "${DEPLOY_USER}@${DEPLOY_HOST}:/opt/app/"
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
- name: Deploy on server
run: |
ssh -i ~/.ssh/deploy_key -p "${DEPLOY_PORT:-22}" \
-o ServerAliveInterval=30 -o ServerAliveCountMax=20 \
"${DEPLOY_USER}@${DEPLOY_HOST}" \
'cd /opt/app && bash deploy/deploy.sh'
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
The design points are explained one by one (all replaced by failed runs):
paths-ignore: Changing the README to also trigger deployment is purely to burn runner minutes and is excluded.**.mdanddoc/**;The workflow file itself is.ymlIt is not affected and can still be verified by changing the pipeline.- Used by Secrets
env:Pass value instead of${{ }}Directly enter the command line. There are two problems with direct splicing: the key value may appear in the log command line; when Secret is not configured, it is rendered into an empty string and the command spells outssh deployer@""Such a weird mistake. I have seen it with my own eyes in the logecho "" > ~/.ssh/deploy_key——It took ten minutes of troubleshooting to realize that Secret’s name was typed incorrectly. Add a null value guard at the beginning, and if it is missing, I will report it to you directly. ServerAliveInterval=30+ServerAliveCountMax=20: Send a keep-alive packet every 30 seconds and tolerate no response for 20 consecutive times. The consequences of not adding this line are shown in Section 7.- rsync
--excludeIt is the lifeline of production data:.env(real key on the server), user upload directory, and database files must be excluded, otherwise--deleteIf it is pushed down, it will be a production accident.
4. deploy.sh: the real deployment logic on the server
put in warehouse deploy/deploy.shcalled by CI SSH, manual upload can also be run:
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
# ---- 互斥锁:两条链路(CI / 手动)同时进来时只许一个跑 ----
exec 9>/tmp/app-deploy.lock
if ! flock -n 9; then
echo "另一个部署正在进行,本次退出"
exit 1
fi
echo "==> 1/4 预拉取基础镜像(重试 3 次)"
for attempt in 1 2 3; do
if docker compose pull redis web umami umami-db; then
break
fi
if [ "$attempt" = 3 ]; then
echo "镜像拉取失败。注意 registry-mirrors 只对 Docker Hub 生效,ghcr.io 需直连或代理站"
exit 1
fi
sleep 15
done
echo "==> 2/4 构建并滚动更新"
docker compose up -d --build
echo "==> 3/4 健康检查(最多 60 秒)"
for i in $(seq 1 12); do
if curl -fs >/dev/null 2>&1; then
echo "健康检查通过"
break
fi
sleep 5
if [ "$i" = 12 ]; then
echo "❌ 服务未恢复健康,容器现场如下:"
docker compose ps
exit 1
fi
done
echo "==> 4/4 ✅ 部署完成"
docker compose ps
Two details that are easily overlooked:
docker compose pullThe service name must be explicitly listed later.. In my composeapi/workeris localbuild:Out, there is no remote mirror, running nakeddocker compose pullIn compose v2, an error will be reported directly. Only four services that actually have remote mirroring are listed.flockPrevent concurrency on the same machine. GitHub ActionsconcurrencyIt only locks the inside of CI and cannot lock the time you run SSH manually, so the mutex at the script level is still necessary.
5. Highlight: Mirror pulling hell and the last mile of ghcr
The first push after the link is configured,Deploy on server Step 1 started frantically refreshing the download log – this brushing lasted for several days: accelerator, changing mirror source, re-running and re-trying in turn. Each time it seemed to be progressing, but the next day it was back to the original state. When I finally clicked Cancel, I realized that I had to first figure out “which machine the download occurred on.” This section is the experience that the whole article wants to convey most.
5.1 Attribute first, then take action
When seeing “Keeping downloading”, the first reaction is easily “GitHub doesn’t work, let’s change to a CI platform”.Calm down and first check which machine the download occurs on. Actions Deploy on server The step is just SSH triggering the remote script, what actually pulls the image isyour serverthe traffic does not go through GitHub at all. Change to cloud effects, Gitee Go, any CI – the server cannot be pulled, and a platform migration is in vain.
There are three diagnostic tools, all executed on the server:
cat /etc/docker/daemon.json # ① 加速器配了没?条目是不是有效 registry 端点?
docker info | grep -A3 'Registry Mirrors' # ② 配了 ≠ 生效了,重启过 docker 吗?
docker pull redis:7-alpine # ③ Docker Hub 链路单独测
docker pull ghcr.io/umami-software/umami:3.4 # ④ ghcr 链路单独测(重点!)
My results at that time: ③ 2 seconds passed through the accelerator, ④ Directly connected to ghcr, it was slow but the layer was moving. This shows that the Docker Hub link has been cured and the problem is concentrated in ghcr.io– This is where most projects get stuck: your analysis/monitoring components (Umami, etc.) are almost all hosted on ghcr.
5.2 Why registry-mirrors cannot save ghcr
daemon.json of registry-mirrors The working principle is: Docker pull Docker Hub Change the mirror when mirroring. For images with full domain name prefix (ghcr.io/xxx、quay.io/xxx、mcr.microsoft.com/xxx)Not valid at allstill directly connected. This is not a configuration problem, but a mechanism problem – so “the accelerator is still slow even if it is equipped with ghcr” is not because you are doing it wrong.
5.3 Layer caching and breakpoint resumption: first separate “stuck” and “very slow”
The performance of ghcr direct connection in China is often: repeated on a certain layer Retrying / Downloading It seems like an endless loop. But it keeps progressing every time it’s rerun:
docker pull ghcr.io/umami-software/umami:3.4 # 中断后原样重跑
# 已完成层显示 Pull complete / Already exists,只补缺的层
The criterion is simple:Check whether the number of bytes of a certain layer changes within several retry cycles.. Let it run until it is moving; if it is completely stationary for more than five minutes, consider changing the route.
5.4 Final plan: Nanjing University ghcr mirror station
The real final word is this——ghcr.nju.edu.cn(Nanjing University mirror station is the agent of ghcr.io):
docker pull ghcr.nju.edu.cn/umami-software/umami:3.4
# 拉下来后改回 compose 里认识的官方名:
docker tag ghcr.nju.edu.cn/umami-software/umami:3.4 ghcr.io/umami-software/umami:3.4
Measured the same Digest: sha256:9f376b984d0b...the content is exactly the same, and the speed has changed from “hanging for days without being able to finish” to minutes. There is no need to change the image name in compose, there is already a local tag with the same name + compose default pull strategy missinga direct hit from the service.
Similar options available:
ghcr.m.daocloud.ioWaiting for university/manufacturer agent station. Agent station availability will fluctuate over time. The idea is Change the registry prefix to pull the same image, and then tag it back to its original name.this trick is applicable to any walled registry.
5.5 A bug that was easily fixed: floating labels
During the troubleshooting, I found that compose says:
image: ghcr.io/umami-software/umami:postgresql-latest # ❌
postgresql-latest The problem with this kind of floating label is two-way: when the upstream releases a new version, it willSilent upgrade(Database migration may not start immediately). You are obviously running 3.4 but cannot nail the version. Confirm the actual running version of the server and then nail it:
docker exec node -p "require('./package.json').version" # 3.4.x
image: ghcr.io/umami-software/umami:3.4 # ✅
Notice:Different tags with the same content do not share local cache–from postgresql-latest nail to 3.4 A real pull will still be triggered, please cooperate with the 5.4 proxy station to do it.
5.6 Preheating methodology: Where does the checklist come from and how to accept it?
Converging all the above experiences into a set of standard actions for “image warm-up before first deployment”:
# ① 清单不要凭记忆列,直接从 compose 提取去重全集:
grep -E '^\s+image:' deploy/docker-compose.yml | awk '{print $2}' | sort -u
# ② 逐条单独 pull,不要串 &&:
docker pull node:22-bookworm
docker pull nginx:1.27-alpine
# ...
# ③ 验收按 "镜像名:tag" 精确比对,不是 grep 个名字前缀:
docker images | grep -E 'node:22-bookworm|umami:3.4|...'
Why so serious? The “fake completion” I’ve stomped on:
- use
&&Six pulls were strung together.The first one failed and the next five failed at all.the terminal looks like “the entire batch has been executed”, and the missing items disappear silently; - use
docker images | grep 镜像名Acceptance, same warehouse with different tags (for examplepostgres:15-alpineandpostgres:16-alpine) hits the entire row as long as either one is present, and the other never pulls down and is completely invisible to you.
In the warm-up listThe easiest thing to miss is building a base image(node:22-bookworm This 1.1GB) – it’s not image: line appears at the top of the service definition, while in build: Dockerfile section FROM inside. Miss it and the deployment will go through download hell all over again during the build phase.
6. The mysterious disappearance of Secrets
workflow After running everything normally for a few rounds, one day Setup SSH Suddenly red, log:
echo "" > ~/.ssh/deploy_key
ssh-keyscan -H "" >> ~/.ssh/known_hosts
Error: Process completed with exit code 1.
${{ secrets.X }} at X If it does not exist, it will be silently rendered as an empty string.“Secret missing” will not be reported. My scenario is that I did a workflow cleanup on the GitHub web page and took away the Repository Secrets (deleting the file itself does not delete the Secrets, but it is easy to get into trouble when cleaning the Actions configuration page).
Two-layer defense (already in Section 3 YAML): Secrets env Pass + The first step is to directly report the null value guard in Chinese. Otherwise you have to start from mkdir The first diary goes back half a day.
7. Broken pipe: Don’t stuff the long build into the CI’s ssh window.
After the mirroring and caching are all in place, there is one last type of accident left:Deploy on server I browsed the build log normally for more than ten minutes (apt installed LibreOffice, pip pulled dependencies, both were domestic sources and were moving forward), and suddenly:
client_loop: send disconnect: Broken pipe
Error: Process completed with exit code 255.
Root cause: The deployment of Actions isA front-end ssh long connectionHang the command from the remote end for more than ten minutes. NAT/firewall devices on cross-border links often have cleaning strategies for “long-term low-traffic TCP sessions”, cutting off SSH and killing the remote build process at the same time.
Two-move combination punch:
# ① ssh 保活(workflow 已加):每 30 秒空包,撑住中间设备
ssh -o ServerAliveInterval=30 -o ServerAliveCountMax=20 ...
# ② 首次全量构建这种重活,绕开 CI,直接在服务器本地后台跑:
cd /opt/app/deploy
nohup docker compose build api worker > /tmp/build.log 2>&1 &
tail -f /tmp/build.log # Ctrl+C 只退出查看,构建不停
All the built layers are cached locally in Docker, and then in CI up -d --build Rebuild only COPY src thin layer——The ssh window is reduced from 15 minutes to 90 seconds, and the opportunity to cut off is gone.. The keepalive parameters are responsible for covering occasionally long builds due to future Dockerfile changes.
八、address already in use :80
The penultimate pit. After everything is ready up -d Report:
Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use
The container port cannot be mapped. It is probably because something on the host has grabbed 80. Positioning in 30 seconds:
ss -tlnp | grep ':80 '
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
My situation is a bit unexpected: I need to update the security system (dnf upgrade-minimal --security), the package manager easily installs nginx as a dependency andStart automatically at bootYes, the host runs nginx naked and occupies 80%, which has nothing to do with the container nginx. Please let it go:
systemctl disable --now nginx
If another detention container occupies the pit,docker rm -f Or find its original compose directory docker compose down. Another troubleshooting experience:use -v parametric docker compose down It will be deleted together with the volume——If the database volume is lost, it is a production accident. Please do not stop the stack during daily operations. -v。
9. Health Check 301 Infinite Loop (nginx Xiaolei)
stacked up but deploy.sh I can’t pass the health check no matter what:curl What is returned is not 200 but a string of 301. What is written in the configuration is:
if ($host != pdf.example.com) { return 301 }
The original intention is to centralize the 301 access of all unofficial domain names to the main domain for SEO weight. but it puts Host: 127.0.0.1 Local health check, LB visit, IP direct visitAll! 301 Reached a domain name that cannot be resolved by this machine at allsuperior. The correct way to write it is to explicitly match only the domain name that needs to be jumped, and let the rest go:
if ($host = example.com) { return 301 }
if ($host = www.example.com) { return 301 }
if ($host != X) and if ($host = X) A pair of unequal conditions, the former is “accidental damage to everything”, the latter is “precision strike”. In addition, for this kind of configuration that “looks normal online”, be sure to confirm that it has been committed into the warehouse – CI deployment is always based on the contents of the warehouse. If you change it locally and don’t push it, it will still be the same online.
10. Closing: Verification Checklist and Experience Summary
Go through the deployment after it turns green:
curl -s # 服务器本机:200 + 依赖状态
docker compose ps # 全容器 Up / healthy
The browser actually measured the core business links (upload, conversion, and background login), and the old container was still in service when the confirmation failed (deploy.sh 3/4 steps are red, which means the old version is still alive).
Finally, the pits of the full text were compressed into oneCheat sheet posted at work station:
| symptoms | root cause | prescription |
|---|---|---|
| Deploy keeps Downloading | The server pulls the overseas registry, which has nothing to do with the CI platform. | Three-level positioning, don’t change platforms |
| Even if the accelerator is equipped with ghcr, it is still slow. | registry-mirrors only Docker Hub | ghcr.nju.edu.cn agent + tag return to original name |
exit 12 |
The server is missing rsync | dnf/apt install rsync |
| pull six strings&& only three strings | If the first item fails, the subsequent items will be skipped silently. | Independent pull item by item, accurate tag acceptance |
echo "" > deploy_key |
Secrets missing rendering empty string | env value + null guard |
exit 255 / Broken pipe |
Long build hangs ssh and is blocked by NAT | ServerAliveInterval + reactivate the server local nohup |
:80 already in use |
dnf update smoothly launched naked nginx | ss -tlnp 定位 + systemctl disable –now |
| Health check returns 301 | $host != Accidentally damaged IP direct visit |
change $host = explicit match |
postgresql-latest |
Silently upgrade and demote floating labels | Check the actual version in the container → nail the label |
docker compose pull Report an error |
The build-only service has no remote mirror | Explicitly list the service name of the image |
Looking back at the YAML of this pipeline, the difference between before and after is actually only a dozen lines; what is really valuable is the one used when troubleshooting.”Attribute first, act later“Muscle memory of “: When downloading is slow, first test the link layered on the target server instead of changing the CI platform first; when error 12 is reported, first check whether the remote command exists, instead of suspecting the network. I hope this record of pitfalls can enable students who are preparing to use the combination of Docker + Actions + domestic servers to get around the pitfalls in one go.
If you have any questions, see them in the comment section, and I’ll reply if I see them.