Self-hosting Next.js and Payload: the maths that works, and three things that break

Serverless pricing is built so that starting costs nothing. It works: a project goes live without an invoice, and the first bills arrive only once something starts to live. The problem is that they grow in proportion not to revenue, but to traffic, headcount and function execution time — three things nobody forecasts at the start.

This article is an arithmetic exercise, not a manifesto. Vercel with a managed database on one side, your own VPS running Coolify on the other. The numbers come from vendor price lists (August 2026), and the three failures described at the end happened to us in production, on exactly the stack that serves the site you are reading.

If you are looking for step-by-step instructions, they live separately: our public starter on GitHub carries a ready configuration of this stack, deployment files included.

What you actually pay each month

The comparison only means something once both sides count the same things: the application, the database, traffic and file storage. Assume a three-person team and 2 TB of traffic a month — the scale of a small content site, not a funded startup.

Two numbers in that table deserve a comment, because they are the ones doing the work. First: traffic. Vercel Pro includes 1 TB and charges 0.15–0.35 USD per gigabyte above it, depending on the region. Hetzner includes 20 TB in European regions — twenty times more, as part of the price. Second: the database. Those 57 USD buy one M10 node, and a production database runs as a three-node replica set, so the real bill is closer to three times that before you add backups and traffic.

There is also an item no table shows: on your own server every additional service is free. A queue worker, your own cron, Redis, a small helper service — same machine. Under per-execution billing each of them gets its own line on the invoice.

What that table does not show

The difference in subscriptions is not a saving; it is a cost moved off an invoice and onto somebody's time. You take on three duties you did not have before: backups together with point-in-time restore, security updates and watching availability. They do not disappear because the bill got smaller — they change owner.

The honest conversion is this: if maintenance takes two hours a month at 150 PLN an hour, that is 300 PLN a month — more than the subscription you saved. Self-hosting pays off when those hours are in the team anyway and cover several projects at once, not one.

Architecture: how this is put together

Since version 3, Payload is not a separate application standing beside Next.js. It runs as a package inside it, sharing the same process, the same build and the same config file. That changes deployment architecture more than it sounds: the admin panel, the API and the frontend are one artefact, so one application container instead of two. What Payload can do as a CMS and when we choose it is covered in: Payload CMS; Next.js itself we take apart in: Next.js.

One container or several

By default we go with one application container plus separate containers for the database, Redis and the proxy. Splitting the application into more processes only makes sense once one of them scales differently from the rest — a worker chewing through a job queue, say, which needs memory during a data import and does nothing for the rest of the day.

Standalone build: 1.5 GB versus 150 MB

Next.js can build a directory containing only what is needed at runtime — without the full tree of development dependencies. It is one line in the config (output: "standalone"), and the difference in image size is an order of magnitude: from roughly 1.5 GB down to about 150 MB. This is not cosmetic. A smaller image means a shorter transfer on every deployment and many times less disk eaten by the successive versions Docker keeps locally.

There is one condition and it is easy to miss: in a multi-stage build you have to copy the static asset and public directories by hand, because standalone does not take them. Skip it and you get a working application with no styles — a symptom that looks like a CSS problem and is a Dockerfile problem.

A docker-compose skeleton

The core of the configuration is below. Coolify adds routing, the domain and the certificate on top, so the file says nothing about Traefik — that is precisely the part you use it for instead of bare Docker.

1services:
2  app:
3    build: .
4    environment:
5      DATABASE_URI: mongodb://mongo:27017/app
6      PAYLOAD_SECRET: ${PAYLOAD_SECRET}
7      REDIS_URL: redis://redis:6379
8    volumes:
9      - media:/app/public/media        # EVERY upload directory, separately
10    depends_on: [mongo, redis]
11
12  mongo:
13    image: mongo:7
14    volumes:
15      - dbdata:/data/db
16
17  redis:
18    image: redis:7-alpine
19    command: redis-server --save 60 1
20    volumes:
21      - redisdata:/data
22
23volumes:
24  media:
25  dbdata:
26  redisdata:
Copy

A note on volumes, because this is the most common mistake. A container filesystem is ephemeral. Every directory the application writes files to must have its own entry under volumes — and “every” means every one, separately for each upload collection. Adding a new collection in Payload without adding its volume is not an error you will see in the logs. You will see it after a restart, when the files are gone.

The complete set — Dockerfile, compose, environment variables and a pre-launch checklist — sits in our public repository: nextjs-payload-starter.

Three things that break

Guides describe the deployment that worked. Below are the three mechanisms that break it most often — each with a real incident from our production as evidence rather than illustration.

1. Memory: image optimisation and the OOM killer

The next/image component processes images inside the application process by default, using the sharplibrary. On a managed platform a separate, scaled service does this and nobody notices. In a container on a VPS with 2–4 GB of memory, uploading a handful of product photos through the Payload panel can summon the OOM Killer — the kernel kills the process to save the system. The application vanishes with no entry in the application logs, because it never got to write one.

The business consequence is out of all proportion to the cause: the site stops responding in the middle of the day, and if a search engine crawler happens to arrive, the page drops out of the index for far longer than the outage itself lasted.

The fix has two steps. Files move off the server into S3-compatible storage — in Payload that is @payloadcms/plugin-cloud-storage with an adapter for Cloudflare R2, AWS S3 or a Hetzner Storage Box. Image processing goes outside too: either to a CDN that transforms on the fly (Cloudflare Images), or by turning off the built-in optimiser (images.unoptimized) and generating sizes on the Payload side at save time. The application then stops holding in memory something it has no reason to hold. More on choosing hosting, a CDN and what actually makes a site faster: Hosting, domains and CDN.

Our version of this bug was worse, because it was quieter. We did not run out of memory — we ran out of volume. One collection's directory had no persistent storage attached, so a container restart took every one of its files while the database records stayed behind, insisting the files existed. It happened twice: once with a media collection, once with the documents collection — eleven report and template files gone from disk while the site went on offering them for download.

2. Cache: ISR has nowhere to live

On Vercel, static page revalidation and revalidateTag work at the level of a global CDN. In your own container the Next.js cache writes to .next/cache on that container's disk. Two things follow, both unpleasant: with two instances each has its own unsynchronised cache, and publishing content in the panel does not clear it by itself. This is, incidentally, one of the prices of headless architecture — more on that in: How headless architecture changes business strategy.

The fix. Your own CacheHandler pointed at from next.config, keeping entries in Redis — then the state is shared across all instances and survives a restart. Plus an afterChange hook in the Payload collections which, after a save, calls revalidatePath or revalidateTag for exactly what changed. The minimal variant — a persistent volume on .next/cache — solves the restart only, not horizontal scaling.

Our version: a disk filled to 89 GB, though nobody had uploaded anything. Bots were scanning addresses that did not exist, and on-demand rendering wrote every one of those responses as a cache file under .next/server/app. It grew for weeks, symptom-free, until the disk ran out. The fix turned out to be one line — a filter that sieves such requests out before they generate a file. The lesson, though, is not about the filter: on your own server the cache is your directory on your disk, and nobody tidies it for you.

3. Resources and state: the build starves the database

If the database sits on the same machine as the application, then docker build takes practically the whole CPU and memory during a deployment. The effect: the site slows down exactly when you are shipping a fix — which is usually when something is already broken. With 8 GB of RAM the Next.js compile can also simply be killed.

• Swap. Four gigabytes minimum. It will not speed the build up, but it will stop it being killed — and that is all anyone asks of it here.
• Building off production. The image is built in GitHub Actions, lands in a registry (GHCR), and Coolify pulls the finished artefact. The production server stops taking part in the build at all — that is the right destination architecture, not an optimisation.
• Separating the database. As the project grows, the database gets its own instance. The first expense worth incurring voluntarily, before an outage forces it.
• Backups with point-in-time restore. A daily dump is not enough if what matters is how much data you can afford to lose. PITR rests on the write log — WAL in Postgres, the oplog in MongoDB — and that log decides whether you go back to yesterday or to ten minutes ago.

Two traps that caught us specifically. First: the build server has no access to the database, so every function generating paths at build time has to account for that — otherwise the build passes locally and falls over in production. The second is sneakier: next build type-checks the whole project, so a directory excluded by .dockerignore can break a deployment while CI — building outside Docker — stays green. That cost us several deployments before we understood that green CI and a successful deploy are two different things.

Certificates. Coolify renews them automatically through Let’s Encrypt, but renewal requires the proxy to answer an HTTP challenge. If Cloudflare sits in front of the server in full proxy mode and the rules do not let everything through, renewal quietly fails and you find out ninety days after the deployment.

When this pays off, and when it does not

The decision is rarely technical. It comes down to whether anyone on the team will pick up the phone when the server stops answering on a Saturday.

If, after this arithmetic, your own server still looks sensible, the starting point is our public starter — Next.js 16, Payload 3, MongoDB and a Coolify deployment in one repository: nextjs-payload-starter.

The wider technology context — what the choice of stack does to the cost of a project — we take apart in: Comparing ways to build a website. And if you want the full maintenance bill, not just hosting: Recurring website fees.

We build deployments like this for ourselves and for clients — see how we build web applications.

Where these numbers come from

Prices come from vendor price lists, as of August 2026: Vercel (20 USD per seat per month, 1 TB of traffic included, 0.15–0.35 USD per GB above that depending on the region), Hetzner Cloud (CPX31: 4 vCPU, 8 GB RAM, 160 GB NVMe, 16.49 EUR a month, 20 TB of traffic in European regions) and MongoDB Atlas (M10 from roughly 57 USD a month per node; a production database is a three-node replica set). The failures described here come from our own deployments, not from the literature.