My Backup Was Crawling Because I Was Using Someone Else's API Quota
The subscription ended, and I had to move a terabyte
The call came in the least convenient way: the OneDrive account holding my nightly backups had no subscription any more, and I had a 1.5-year Google subscription sitting unused. Move the backup, keep the history, don’t lose a night.
The first thing I hit was not a technical problem: I could not remember how the backup was wired. rclone? Duplicati? Something Synology shipped? On a three-host estate with 78 compose stacks — the one in Running Production Infrastructure Solo — “which tool writes my backups” is a question I should have been able to answer from memory and could not. So step one was reading the machine.
What was actually there
Two rclone configs, which is the kind of detail that bites: the GUI config that Backrest (the restic front end) uses, plus a separate stale CLI copy that a cron job or a human shell had been using at some point. Change one and you have changed half the system.
The migration itself is boring when done in the right order, and I did it read-first:
rclone.conf append [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf (CLI copy) same two remotes
Portainer stack 177 env → the new crypt remote; config :ro → :rw
restic init new repo on Google Drive
Backrest SetConfig dsm-repo URI → the new locationBefore touching the plan URIs I checked one property in the source, because
everything depended on it: a restic repository’s identity is
guid = sha256(id) — a stable value derived from the repo id, not from
its URL. So repointing a plan at a new backend cannot silently fork the
history. All six plans came out byte-identical to the originals, and the
repo id/guid were untouched.
Then the seed stopped moving, and the log named a stranger
The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed to a crawl. The live log had the answer:
'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644 <- not my project
quota_limit_value: 840000202264815644 is rclone’s own shared OAuth project. My terabyte seed
was drawing on a per-minute read quota pooled with every rclone user on
earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my
disk and my link were all idle. By the time I measured, the seed was fully
stalled — 0 bytes in 90 seconds, 159 errors, still rateLimitExceeded —
and the six nightly plans had all failed with the same wall:
Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
and limit 'Requests per minute' ...“Slow” and “blocked” look similar in a progress bar and are completely different problems. Ninety seconds of sampled zeros is what made the call.
You need your own OAuth client — and the console is unavoidable
There is exactly one real fix: stop using the shared client and bring your own. That means a Google Cloud project, an OAuth consent screen, and a credential. Two settings cost real time:
- Application type = Desktop app. rclone’s redirect is
http://127.0.0.1:53682, and that client type permits it without extra configuration. - Publish the app. Leave it in “Testing” and Google expires your refresh token every 7 days — which turns “working” into a silent weekly outage.
I checked whether I could shortcut the console with the CLI that was already
authenticated here. I could not, and the reason is worth knowing: gcloud
was signed in only as a service account in an unrelated project, and
service accounts cannot create consent screens or OAuth client IDs. The
credential has to belong to a project a human owns. It was a dead end, and I
verified that instead of asserting it.
Three traps in the authorization flow
- Publishing an unverified app can hard-block a restricted scope.
.../auth/driveis restricted, and an app that is published without completing Google’s verification review is sometimes blocked outright with no bypass. In my case the bypass existed — the “Google hasn’t verified this app” screen, then Advanced → Go to rclone-backup (unsafe) — but that is a coin flip on scope and review state, not a guarantee, and “unsafe” is only a label meaning the app never went through a paid audit. - The callback accepts exactly one request. A reload, or a second tab,
hits
127.0.0.1:53682with no code and kills the attempt. My first try died exactly that way (no code returned), and it looks like a credentials problem. One tab. Don’t reload it. - Finish the consent screen on the machine that runs rclone. The
redirect goes to
localhost, so a phone or another PC cannot complete it — the browser has to be the one on the same host as the process waiting on that port.
Then: read the token, write client_id + client_secret + token into
both configs (the GUI one and the stale CLI copy), and restore the two
flags that matter for a long unattended upload: use_trash = false and
stop_on_upload_limit = true.
The old side: verify by reading, not by asking
Purging the old remote reported an error — itemNotFound — and the space
came back anyway:
Used: 1.012 TiB -> 3.800 GiB
Free: ~0 -> 1.007 TiB
DSM folder: present -> gone Trashed: 0 BThe error was rclone complaining after the work finished. Also worth
recording: the tool’s documentation says personal accounts cannot
permanentDelete, which would have meant emptying the recycle bin by hand —
in practice the trash was already empty and roughly 1 TB was reclaimed with
no browser step. Documentation describes the API; the API is not always what
you get.
One more measured thing, because it changed my next decision: pausing the file-sync container that shared that machine halved the load average — 16.70 → 8.89 — which is the difference between “the network is slow” and “the box is busy”.
What I would do differently
- Own the OAuth client from day one. A shared client id is a queue you are standing in, with no status page and no one to call.
- Answer “what writes my backups” from a document, not from memory. Two rclone configs is a normal amount of drift for a system nobody has drawn in a year.
- Read the destination, not the exit code. The purge “failed” and freed a terabyte; a successful-looking rename can land nothing. Same lesson as It Reported Success. Nothing Had Changed. — this estate keeps teaching it.
- Never start a multi-day upload while the last few percent of your
attention is elsewhere. Checking for
RATE_LIMIT_EXCEEDEDat 0.27% costs minutes; discovering it after two days costs the window.
The result, honestly stated
The plumbing works: the plans live on Google Drive, the repo identity is unchanged, the seed is now spending my quota instead of a shared one, and the old side gave back about a terabyte. What it is not, yet, is a backup: at the time I wrote this, the first seed had uploaded 2.88 GB of ~1075 GB and the new repo had zero snapshots — nothing is restorable until the first snapshot lands. That is the number I watch, because “the transfer is running” and “you can restore” are different claims.
