Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day

I run an automated monitoring pipeline that watches Carousell for newly-listed items and pings me on Telegram when something fresh appears. It is one container on my Synology NAS, driven through Portainer, with a single Python process as PID 1 on a Docker bridge network. One day it stopped pinging. The container was healthy, the scraper was still archiving listings into NocoDB — but every notification died with the same three characters in the log:

telegram sendPhoto failed: 400 ... nginx/1.30.1

That string, nginx/1.30.1, became the villain of a day-long debugging session. This is the story of how a deceptively small bug turned out to be three separate problems stacked on top of each other — and how I’d find each one faster next time.

The problem

The bot had two jobs: collect listings, and notify me. Collection worked. Notification didn’t. The log said 400 Bad Request and pointed at a server identifying itself as nginx/1.30.1.

My first assumption was textbook: something in the network is intercepting the request. nginx/1.30.1 is not a header I associated with Telegram’s API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled like a local reverse proxy or a VPN gateway rejecting traffic before it ever left the machine.

Assumptions like that are cheap. They’re also wrong a lot. Let me walk through what I actually found.

What I tried, and why each thing failed

Round 1: it must be DNS / IPv6

I checked getMe from inside the container. It worked. Then I checked sendMessage. It failed. Same token, same container, one method up, one down.

The giveaway was in how the two endpoints resolved:

# inside the container
$ getent hosts api.telegram.org
2001:67c:4e8:f004::9   api.telegram.org

The container resolved api.telegram.org to IPv6 only — a single AAAA record and no A record. The container ran on a Docker bridge network with no IPv6 connectivity. So every request that needed api.telegram.org was trying to reach an IPv6 address it had no route to.

The fix looked obvious: pin the IPv4 address with extra_hosts. I did, and the DNS answer became exactly what I expected:

149.154.167.220   api.telegram.org

And the notifications still failed. Same 400. My proud fix did nothing.

Round 2: I picked the wrong IP

Here’s a mistake worth remembering: I hard-coded 149.154.167.220 from memory. When I finally queried a public resolver, the real A record for api.telegram.org was different:

149.154.166.110   api.telegram.org   # what DNS actually says
149.154.167.220   api.telegram.org   # what I hard-coded

167.220 is inside Telegram’s IP range (149.154.160.0/20), so it’s not “wrong” in the sense of being somebody else’s server — but it’s not the active endpoint, and bot-API traffic to it behaved unpredictably. I switched to 166.110. It still failed. So the IP wasn’t the whole story either.

Round 3: the same command, two different outcomes

This was the moment the whole thing got weird. Inside one container, at almost the same second:

• the long-running monitor process (PID 1) → sendPhoto failed, 400
• a fresh docker exec process running identical code → sendPhoto succeeded, 200

Same container. Same code. Same payload. Same second.

I ruled out token, chat_id, emoji in the caption, Unicode quotes, image format, environment variables, and the image URL itself — I download-tested the photo and it was a perfectly valid JPEG. Nothing in the code differed between the two paths.

When you hit a contradiction like that — “identical inputs, opposite outputs” — the answer is usually that the inputs aren’t identical. In my docker exec test, I had tested against a lightweight sample image; but the PID 1 loop was processing real scraped listing photos where binary byte sequences happened to break multipart boundary parsing. The binary payloads were not the same. But finding the difference took one more frame change.

Round 4: nginx/1.30.1 is Telegram

I re-read the raw 400 response instead of assuming. The response was an HTML “400 Bad Request” page, and the Server header was nginx/1.30.1.

That nginx is Telegram’s own edge. Telegram’s API sits behind nginx (version 1.30.1), and when a request is malformed before it reaches their application layer, nginx itself answers with a plain 400 page — no JSON, no friendly error code.

That reframed everything. The request wasn’t being intercepted. It was reaching Telegram, and Telegram’s nginx was rejecting it as malformed.

And the malformed thing was the one part I hadn’t scrutinized: the sendPhoto multipart body.

The fix

My original helper assembled the multipart body by treating headers and binary payloads as a list of “lines” and joining them with b"\r\n".join(body_lines) — the kind of “obviously fine” shortcut that ships a framing bug:

# BROKEN: pre-terminated chunks or line-split binary data re-joined with b"\r\n"
body = b"\r\n".join(body_lines)

It fails in two ways, and neither is bytes.join() rewriting the JPEG — b"\r\n".join() only inserts the delimiter between list elements and never touches bytes inside an element. The damage happens upstream. First, if the payload ever passes through line-oriented handling such as .splitlines(), every lone 0x0D or 0x0A byte inside the compressed image stream is treated as a line break and rewritten to \r\n, which mutates the JPEG and desynchronises the Content-Length that was computed from the original file. Second, if any chunk in body_lines already carries its own trailing \r\n, the join adds a second one, so the body reads --boundary\r\n\r\nContent-Disposition: ... — and in MIME framing a blank line terminates the part headers, so that part becomes an anonymous body whose first line is the Content-Disposition header. Telegram’s edge nginx/1.30.1 rejects that as malformed with an HTML 400 long before the Bot API parses anything.

That also closes the Round 3 contradiction. The docker exec one-liner I typed by hand sent a minimal two-field payload from the freshly edited script, while the long-running PID 1 process was still executing the old in-memory framing path against a real marketplace JPEG whose bytes were long enough to hit the doubled-\r\n case — same function, different input, one 200 and one 400.

The correct approach is to frame every part explicitly and concatenate the byte chunks verbatim with b"".join(parts):

boundary = "----tg" + token_hex(8)
parts = [
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="chat_id"\r\n\r\n',
    f"{chat_id}\r\n".encode(),
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="caption"\r\n\r\n',
    f"{caption}\r\n".encode(),
    f"--{boundary}\r\n".encode(),
    b'Content-Disposition: form-data; name="photo"; '
    b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n',
    image_bytes,                   # ← binary stays untouched
    b"\r\n",
    f"--{boundary}--\r\n".encode(),
]
body = b"".join(parts)             # concatenate raw byte chunks

The difference is b"".join(parts) instead of b"\r\n".join(...): each part is already a complete chunk with its own framing, and we concatenate them verbatim rather than inserting a delimiter between every element.

There’s a second, higher-level fix worth naming: stop asking Telegram to download the image for you. My earlier code passed the image as a URL and let Telegram fetch it:

tg("sendPhoto", {"photo": image_url, "caption": caption})

That means Telegram’s servers have to reach out to the source CDN — and if that CDN is flaky or geo-blocks Telegram’s crawlers, you get intermittent failures that are impossible to reproduce locally. Downloading the image yourself, then uploading the bytes, removes an entire class of flakiness and gives you a chance to validate the bytes before you send them.

What I’d do differently

1. Read the raw response, not the assumption. nginx/1.30.1 sat in logs for hours while I chased phantom proxies. Look up who owns the header before inventing a man-in-the-middle.
2. Verify a hard-coded IP against a live resolver. Memory is not DNS. dig +short api.telegram.org takes three seconds and would have saved me a whole round.
3. Treat “identical inputs, opposite outputs” as a lie. The inputs were never identical — the two processes differed somewhere I wasn’t looking. The honest move is to diff the bytes, not the code.
4. Never hand-join binary data with a text delimiter. b"\r\n".join() only inserts a delimiter between list elements — so any element that already ends in \r\n gains a second one, and any binary stream that went through .splitlines() is already rewritten. Frame each part yourself, then use b"".join().

The result

After the multipart fix, the monitor pushed a real notification through on the next tick — image and all. I also folded in a retry queue so a failed notification stays marked un-sent and is retried 30 seconds later instead of silently vanishing. The whole episode cost the better part of a day, but it turned into exactly the kind of bug I’ll never trip over twice.

This monitoring pipeline was originally built for high-frequency internal inspection, but the troubleshooting lessons apply to any production service that relies on outbound notifications. If you’ve got a monitor, a scraper, or an alert pipeline that needs to reach people reliably, let’s talk.

I build reliable notification and monitoring pipelines — Telegram/WhatsApp alerts, scrapers, and the infrastructure around them. Tell me what you need to watch and where you need it delivered:

• WhatsApp me →
• Email me →
• hoelee.com

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

LinkedIn GitHub Gitea [email protected]

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts