toolinghttpdebugging

A love letter to cURL — the world's most underrated debugger

Postman is great. Insomnia is great. But the moment something goes wrong in production, the tool I reach for first is the one that's been on every Unix box since 1996.

· 7 min read

I want to start with a confession. For about a decade of my career, I used curl the way most engineers do: to copy-paste the contents of the “Copy as cURL” button from Chrome DevTools into a shell, run it once, see what came back, and never think about the tool again.

I want to start with a confession because, looking back, that was an embarrassing waste of curl. Sort of like owning a piano and only playing the C-major scale with one finger. There’s a whole instrument in there. Most engineers I know — including, until recently, me — don’t realise it.

This is a long-overdue thank-you post to the most underrated debugger in our profession.

A brief, biased argument

I have nothing against GUI HTTP clients. Postman has, on net, probably made the world a better place for the average web developer. Insomnia has a nicer UI than Postman. Bruno is open source and ships faster than either. Hoppscotch runs in a browser. These are all good tools and they all have their place.

Their place is exploration. Building a request from scratch. Sharing a collection with the rest of the team. Onboarding a new hire. Generating client code from a spec. Those are real, valuable workflows and a GUI is the right shape for them.

The place a GUI is wrong for is debugging. Debugging happens late, on someone else’s box, over an SSH session, when something is already on fire and the GUI is not installed and you don’t have the patience to install it. In that mode, the tool that wins is the one that’s already there.

curl is already there. It is, in fact, the most “already there” piece of software on the modern internet. There is curl in your container. There is curl in your Raspberry Pi. There is, somewhat absurdly, curl running on Mars right now. It will outlive you. You should probably learn it properly.

The six flags that get you 80% of the way

Memorise these. They go together.

curl -sS \
     -i \
     -X POST \
     -H 'Content-Type: application/json' \
     -H 'Authorization: Bearer eyJhbGciOi...' \
     --data '{"hello":"world"}' \
     https://api.example.com/v1/things

What’s going on:

  • -s makes curl shut up about the progress meter. -S puts errors back on stderr. The combination — -sS — is what you want approximately always. Without -S, a silent failure is genuinely silent. You will, at some point, lose half an hour to this. Save yourself.
  • -i includes the response headers in stdout. They are, more often than not, the part of the response you actually care about. Run curl without -i and you’ll spend the next ten minutes wondering why your authenticated endpoint returns an empty body. It’s because it’s returning a 307 and you can’t see it.
  • -X POST does what it says. There’s -d/--data for the body. There’s also --data-raw, --data-binary, --data-urlencode, and a couple of others, each for a slightly different shape of payload. The mnemonic I use: --data is for JSON, --data-binary is for the moment you discover that JSON is corrupting your bytes.

That’s the boring 80%. The fun starts past it.

When you need to know what actually went over the wire

The single most-useful flag in curl and almost nobody talks about it:

curl --trace-ascii /tmp/trace.txt https://api.example.com/healthz

This writes every byte sent and received, in a readable form, to /tmp/trace.txt. Not “the headers”. Every byte. Including the bytes that are causing you to lose your mind.

A non-exhaustive list of bugs I have personally solved by cat-ing a --trace-ascii output and squinting at it:

  • A reverse proxy that was quietly stripping Transfer-Encoding: chunked and breaking streaming uploads in production. The proxy config was managed by another team. The bug had been live for about six weeks. We found it in twenty minutes.
  • An HTTP client library appending an extra \r\n to a JSON body, which the server’s body parser silently treated as “end of body, but the body is invalid, return 400 with no detail.”
  • A load balancer rejecting HTTP/2 in production but accepting it in staging. Root cause: a misconfigured ALPN cipher list, six letters different. The “X-Service-Version” header was right. The TLS handshake metadata was not.
  • An “invalid signature” error caused by a single zero-width space pasted from someone’s Slack message into a config file. I am not making that up. I wish I were.

I want to be clear here: I could not have caught any of these in Postman. Postman would have shown me “400 Bad Request” and a body that said “invalid request”. --trace-ascii showed me the bytes. The bytes were lying. The bytes are always lying. You just have to look at them.

The timing breakdown

This is the one I tell every junior engineer about and they look at me like I just handed them a cheat code, which, in fairness, I sort of did. Add this to your ~/.curlrc:

# ~/.curlrc
-w '\n
namelookup:     %{time_namelookup}s
connect:        %{time_connect}s
appconnect:     %{time_appconnect}s
pretransfer:    %{time_pretransfer}s
starttransfer:  %{time_starttransfer}s
total:          %{time_total}s
http_code:      %{http_code}
size_download:  %{size_download} bytes\n'

Now every curl invocation tells you, for free, exactly where the time went. DNS slow? That’s namelookup. TLS handshake slow? That’s appconnect. Server slow? That’s starttransfer minus pretransfer. Egress slow? Subtract.

If you’ve ever spent half an hour in a meeting arguing about whether your latency problem is “the network” or “the application”, and the answer turned out to be “TLS handshake on a cold connection”, this is the tool that would have ended that meeting in 30 seconds.

Replaying traffic, badly, with shell

Most teams I’ve worked with have at least once gone through the ritual of “we need a load testing tool”. They evaluate k6, locust, vegeta, gatling, sometimes JMeter (don’t), pick one, document something on Confluence, and then nobody ever uses it again because running it requires too much setup.

curl plus xargs plus a directory of captured requests will, in my experience, cover 70% of the use cases of an actual load testing tool, with zero setup:

# Replay 1000 captured requests at staging, 20 in parallel,
# count response codes.
ls captured/*.json | head -1000 |
  xargs -I{} -P20 \
    curl -sS -o /dev/null -w '%{http_code}\n' \
         -H 'Content-Type: application/json' \
         --data-binary @{} \
         https://staging.api.example.com/v1/things |
  sort | uniq -c | sort -rn

I have used variants of that one-liner to:

  • Confirm a new validation schema rejects ~0.03% of historical traffic, none of which integration tests had ever caught.
  • Bisect which day of which release introduced a sudden 403 spike (it was a Tuesday, the spike was a customer with a leading whitespace in their API key, the customer’s lawyers were involved).
  • Smoke-test a new region in a CDN before swinging DNS, by combining the above with --resolve (see next section).

I’m not saying don’t use a real load tester for real load testing. I’m saying: a surprising amount of “real” load testing is actually this, with a fancier wrapper.

TLS is the rest of curl’s superpower

There is a whole second curl hiding inside curl, and it’s about TLS. A non-exhaustive tour:

# What certificate chain is the server actually serving?
curl -vI https://api.example.com 2>&1 | grep -E '^\*'

# Force a particular TLS version, to see if the new version is the
# problem or the old version is.
curl --tls-max 1.2 https://api.example.com

# Mutual TLS, in two flags.
curl --cert client.pem --key client.key https://api.example.com/private

# Pin a hostname to a specific IP, bypassing DNS, GeoDNS, Anycast,
# and whatever other routing voodoo is between you and the server.
# This is the trick.
curl --resolve api.example.com:443:203.0.113.42 https://api.example.com

That last one — --resolve — is the killer feature. The first time you use it to hit one specific edge node in a CDN, in a specific region, bypassing every routing layer between you and that one box, you will wonder how you ever debugged a CDN issue any other way. (Answer: very, very slowly, while arguing with support.)

The real argument

You could do all of the above in a GUI. You could also, in principle, drive a screw with a butter knife. The question is whether the tool fits the situation.

Debugging fits a specific situation. The situation is: tired, late, on someone else’s machine, with no patience for friction. In that situation, the tool that wins is the one that’s already installed, scriptable, pipeable, honest about what’s happening at the byte level, and old enough that the answer to any question you have is on Stack Overflow.

curl is all four. It has been all four since 1996. It will still be all four in 2050. It works in a Docker FROM scratch image (well, with the static build). It works over SSH on a box that hasn’t been updated since 2014. It works on Mars.

Spend a weekend with man curl. Yes, the whole thing — it’s not that long, and most of it is genuinely useful. Take notes. Build yourself a cheat sheet, preferably on actual paper. The next time something is on fire at 3am, you’ll save half an hour and possibly your sanity.

Daniel Stenberg, if you’re reading this: thank you. Genuinely. We all owe you several beers.