Avoiding PostgreSQL database corruption

TL;DR: Don't ever set fsync=off, don't kill -9 the postmaster then delete postmaster.pid, don't run PostgreSQL on network file systems.

Reports of database corruption on the PostgreSQL mailing list are uncommon, but hardly rare. While a few data corruption issues have been found in PostgreSQL over the years, the vast majority of issues are caused by:

• Administrator action;
• Misconfiguration; or
• Bad hardware

A recent mailing list post asked what can be done to reduce the chance of corruption and keep data safe.

---

If you think you have a corrupt PostgreSQL database, stop the database server and take a complete copy of the data directory now. See the Corruption page on the wiki. Then ask for help on the pgsql-general mailing list, or for critical/urgent issues contact a profesional support provider.

Do not attempt to fix the problem before taking a complete copy of the entire data directory. You might make the problem much worse, turning a recoverable problem into unrecoverable data loss.

---

Here's my advice for avoiding DB corruption issues, with some general PostgreSQL administration advice thrown in for good measure:

Read the manual

Read the reliability section of the manual in detail.

Many of the issues people on the mailing lists ask about could be avoided if they'd only read the manual. The PostgreSQL manual is well written, comprehensive, and will teach you about SQL and databases in general, not just PostgreSQL. Reading it cover-to-cover helped me learn more about database systems than any one other thing I've ever done. Read it.

PostgreSQL is easy enough to use, so you don't have to read the manual, but you'll get a lot more out of the system if you do. You'll be able to tune it for better performance, know how to avoid a few pitfalls, and know how to use more of its features. You'd know to avoid using SIGKILL (kill -9) to terminate the postmaster, for example.

Updates

Bug fixes are no good if you don't apply them. Keep up to date with the latest PostgreSQL patch releases. Don't be one of those people still running 9.0.0 when 9.0.10 is out. The team releases updates for a reason. Minor point releases require minimal downtime as there's no need to dump and reload; just stop the DB, install the new binaries, and start the DB. Even this downtime can be avoided with failover and failback using something like repmgr.

As per the versioning policy they're extremely conservative about what gets back-patched into released major versions so upgrades are very safe. Read the versioning policy. Just read the release notes for each version in between yours and the new version, then install it.

For more on this topic read the FAQ entry A bug I'm encountering is fixed in a newer minor release of PostgreSQL, but I don't want to upgrade. Can I get a patch for just this issue?

Keep an eye on the end of life dates. Plan to upgrade to a new major version well before your version goes EOL.

Administrator action

Don't kill -9 (SIGKILL) any PostgreSQL processes. It should be fine, but it's still not smart. kill -9 should be your last resort; first use pg_cancel_backend then pg_terminate_backend (which is effectively a kill -15). If you kill -9 a backend the whole DB will terminate and restart, breaking all connections and forcing crash recovery.

Never delete postmaster.pid.

(Corruption should not actually occur unless you kill -9 the postmaster but leave other postgres processes running, then delete postmaster.pid and start a new postmaster. If you insist on unlocking the gun safe, loading the shotgun, aiming it at your foot and pulling the trigger we can't stop you. Don't do that.)

Don't delete anything in the data directory, with the sole exception of the contents of pg_log if your logs are within your data directory. Everything else is vital and should never be messed with directly by the administrator. Do not delete anything from pg_clog or pg_xlog, these are vital components of the database system.

Do not run your database with fsync = off if it contains any data you care about. Turning off the fsync configuration parameter effectively gives PostgreSQL permission to avoid slow syncs at the risk of severe and unrecoverable data corruption if the server crashes or loses power. If it's possible to recover such a database at all it'll take many hours of time from a serious (and expensive) expert. These days there's rarely a reason to turn fsync = off anyway; you can achieve many of the same benefits without the corruption risks using asynchronous commit and a commit delay.

The system catalogs (pg_catalog.*) are there for looking, not for touching. Do not ever INSERT, UPDATE or DELETE in the system catalog tables unless you are absolutely certain you know what you are doing. Only a superuser or (for some catalogs) the database owner can mess directly with the catalogs, and you shouldn't be running as either day-to-day.

If you need pg_resetxlog then you already have a corruption problem. pg_resetxlog is a last resort for getting a damaged database going again, not a semi-routine operation like MySQL's myisamchk. If you have to pg_resetxlog a DB, you need to work out what went wrong and why, then dump the DB(s), re-initdb, and restore the dumps. Don't just keep on using the damaged database.

Backups, replication and PITR

The backups section of the manual is good. Read it. You should also look at the Barman tool, which helps automate many of these tasks.

Maintain rolling backup dumps with proper ageing. For example, keep one a day for the last 7 days, then one a week for the last 4 weeks, then one a month for the rest of the year, then one a year. Backups won't stop corruption or damage happening, but they'll help you recover with a minimum of pain. In my view good old pg_dump is the tool of choice here; I tend to produce one -Fc dump per database and a pg_dumpall --globals-only for users/roles/etc.

Use warm standby with log shipping and/or replication to maintain a live copy of the DB, so you don't have that painful window between your last backup and when the failure occurred. Do not rely solely on replication, take dumps too.

If you want point-in-time recovery, keep a few days or weeks worth of WAL archives and a basebackup around. That'll help you recover from those "oops I meant DROP TABLE unimportant; not DROP TABLE vital_financial_records;" issues with a minimum of data loss.

Barman can help with warm standby and PITR.

If taking filesystem-level backups of the data directory, you don't have to stop the DB if you use pg_basebackup, or use pg_start_backup and pg_stop_backup correctly. See the manual. You do have to include the whole data directory. Do not omit folders you think are unimportant from the backup. People who left pg_xlog or pg_clog out of their backup as "unimportant" periodically turn up asking for help on the mailing list. Don't be one of them.

Remember to test your backups, don't just assume they're fine.

Hardware

Plug-pull test your system when you're testing it before going live. Put it under load with something like pgbench, then literally pull the power plug out so your server loses power and shuts down uncleanly. If your database doesn't come back up fine you have hardware, OS or configuration problems. Do this repeatedly while under load, until you're confident it'll come up reliably.

Use good quality hardware with proper cooling and a good quality power supply. If possible, ECC RAM is a nice extra.

Never, ever, ever use cheap SSDs. Use good quality hard drives or (after proper testing) high end SSDs. Read the SSD reviews periodically posted on this mailing list if considering using SSDs. Make sure the SSD has power-loss protection for its write cache - a supercapacitor/ultracapacitor, low-power capacitor array, secondary battery, or other write-cache protection. Always do repeated plug-pull testing when using SSDs, don't just trust the specifications.

Avoid RAID 5, mostly because the performance is terrible, but also because I've seen corruption issues with rebuilds from parity on failing disks. If you're using spinning disks RAID 10 is a better option for a database.

If you have a hardware RAID controller with a battery backed cache (BBU), test the BBU regularly. A battery backup that doesn't work is no good, and a RAID controller in write-back caching mode without a working BBU is guaranteed to severely corrupt your data if the system loses power.

Use a good quality hardware RAID controller with a battery backup cache unit if you're using spinning disks in RAID. This is as much for performance as reliability; a BBU will make an immense difference to database performance. Be aware that if your RAID controller fails you must be able to get a hold of a compatible one before you can read your array. If you don't add a battery backed cache you won't gain much over Linux's md software RAID, which has the advantage that arrays are portable from machine to machine. Do not use cheap on-motherboard/in-BIOS "RAID" systems, they combine the worst features of hardware and software RAID.

If you're going to have a UPS (you shouldn't need one as your system should be crash-safe), don't waste your money on a cheap one. Get a good online double-conversion unit that does proper power filtering. Cheap UPSs are just a battery with a fast switch, they provide no power filtering and what little surge protection they offer is done with a component that wears out over time and load, becoming totally ineffective. Since your system should be crash-safe a cheap UPS will do nothing for corruption protection, it'll only help with uptime.

I shouldn't have to say this, but ... don't run your database off a USB flash drive ("USB memory key" or "USB stick").

Software and OS

Use a solid, reliable file system. zfs-on-linux, btrfs, etc are not the right choices for a database you care about. Never, ever, ever use FAT32.

If on Windows, do not run an anti-virus program on your
database server. Nobody should be using it for other things or running programs on it anyway. Antivirus software intercepts system calls in all sorts of fun and interesting ways. Many of these are subject to exciting race conditions and bugs when software like PostgreSQL has several processes all reading from and writing to the same files. Antivirus software may also try to "clean" the database files, causing severe corruption. If you absolutely must run antivirus software, add exclusions for the PostgreSQL data directory and PostgreSQL install directory. If your AV software supports process exclusions, add an exclusion for all programs in the PostgreSQL bin directory, especially postgres.exe.

Make sure your OS is disabling your hard drive's write cache or properly flushing it. These days this is a pretty safe bet unless you're using a cheap SSD that ignores flush requests. Plug-pull testing should catch write-cache issues, as should the tools mentioned in the reliability documentation linked to above.

You may also wish to consider reading these wiki articles:

• PostgreSQL FAQ
• Reliable writes
• Direct storage vs SAN
• SCSI vs IDE/SATA disks
• Installation and administration best practices