Unleashing Efficiency: A Step-by-Step Guide to MongoDB Backup with LVM Snapshots

 

NOTE:

• As part of this demo, we are assuming that you have already installed Mongodb  Enterprise server, if not kindly check How to install Mongodb on Redhat\Centos Linux using Tarball .tgz?  to get it installed.

• Be aware the steps which is articulated in this article is intended to demonstrate how to backup mmongodb using LVM based snapshot. It's always best to test in a non-production environment and consider the trade-offs before proceeding with this technique.

Introduction:

• Ensuring the safety of your MongoDB data is a top priority for any database administrator. Regular backups play a crucial role in safeguarding critical information and minimizing the risk of data loss.

•  In this blog post, I will guide you through a straightforward and efficient method for creating MongoDB backups using LVM (Logical Volume Manager) snapshots and automate the overall process using bash script. This approach allows you to capture your database's state without affecting its performance while ensuring data integrity.

Step 1: Verify Disk and Volume Group Details

Before proceeding with the backup process, the first step is to validate the raw/partitioned disks and review the volume group configuration. Use the following commands to display important information:

• `pvdisplay`: This command is used to display information about physical volumes (PVs) on a Linux system that are used in the LVM (Logical Volume Manager) configuration.

[root@mongodb2 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               vg_mongo
  PV Size               10.00 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              2559
  Free PE               767
  Allocated PE          1792
  PV UUID               Blggdd-LG1H-GPwP-vxhv-UHJf-r8Zc-XAxGyv

The physical volume /dev/sdb1 belongs to the volume group vg_mongo, and its total size is approximately 10.00 GiB.

• `vgdisplay`: This command provides comprehensive information about the volume groups present in the LVM configuration. It includes details such as the volume group's size, physical volumes it contains, the number of logical volumes within the group, and other relevant metadata.

[root@mongodb2 ~]# vgdisplay

  --- Volume group ---
  VG Name               vg_mongo
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  55
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <10.00 GiB
  PE Size               4.00 MiB
  Total PE              2559
  Alloc PE / Size       1792 / 7.00 GiB
  Free  PE / Size       767 / <3.00 GiB
  VG UUID               rNKvfc-SVdE-1pJf-7oAd-iZJV-QCxw-xC3Dh5

The volume group vg_mongo is the root volume group where our MongoDB deployment and snapshots will be stored. It uses the LVM2 (Logical Volume Manager version 2) format and currently consists of one physical volume (/dev/sdb1). The total size of the volume group is approximately 10.00 GiB.

• `lvdisplay`: This command provides comprehensive information about the logical volumes present in the LVM configuration. It includes details such as the logical volume's size, allocation, status, and other relevant metadata.

[root@mongodb2 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg_mongo/lvmongo
  LV Name                lvmongo
  VG Name                vg_mongo
  LV UUID                Op3YOJ-edO0-i9ba-h2bR-95cU-42aX-xaybRR
  LV Write Access        read/write
  LV Creation host, time mongodb2.dbabench.com, 2023-07-20 11:37:22 +0800
  LV Status              available
  # open                 0
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

  

The device path of the logical volume lvmongo in the volume group vg_mongo is /dev/vg_mongo/lvmongo, and its total size is 5.00 GiB. This logical volume lvmongo is where I have deployed my MongoDB.

Step 2: Check Partition and Mount Status 

Verify the status of the partitions and their corresponding mount points by executing:

$ lsblk
$ df -hTP /mnt/mongo/

• `lsblk`: Lists all block devices on your system, helping you verify the partition layout.

• `df -hTP /mnt/mongo/`: Displays disk usage and filesystem information for the `/mnt/mongo` mount point, ensuring the correct setup. Where the `/mnt/mongo` is the mount point where my lvm lvmongo is mounted.

Step 3: Lock the Database and Capture Oplog Information

• To ensure data consistency during the backup process, lock the MongoDB database using the following commands:

$ mongo -u<username> -p<password> --port <port> --quiet fsynclock.js

$ cat fsynclock.js
// fsynclock.js
db.fsyncLock();

• The script locks the MongoDB instance for write operations, ensuring no changes occur during the backup process.
• Next, capture the Oplog (operation log) information using the `backup_oplog_ts.js` script. 

$ mongo -u<username> -p<password> --port <port> --quiet backup_oplog_ts.js > oplog.pos

cat backup_oplog_ts.js
-----------------------
// backup_oplog_ts.js
var local = db.getSiblingDB('local');
var last = local['oplog.rs'].find().sort({'$natural': -1}).limit(1)[0];
var result = {};
if(last != null) {
    result = {position : last['ts']};
}
print(JSON.stringify(result));

• This script fetches the latest Oplog timestamp from the MongoDB instance and saves it to the `oplog.pos` file. The Oplog is essential for point-in-time recovery during restores. 

Step 4: Take the LVM Snapshot

• Create an LVM snapshot of your MongoDB LVM volume using the following command:

$ lvcreate -L500M -s -n mongosnap_20July2023 /dev/vg_mongo/lvmongo

Explanations:

- `lvcreate`: Allocates a new logical volume as an LVM snapshot with the specified size (500 MB) and name.

- `-s`: Specifies that the new logical volume should be a snapshot.

- `-n mongosnap_20July2023`: Names the LVM snapshot as "mongosnap_20July2023".

- `/dev/vg_mongo/lvmongo`: Specifies the original LVM volume (mongo deployment) to create a snapshot from.

Please make sure the volume group contain sufficient free space and required snapshot size to hold the data .

Step 5: Mount the Snapshot for Archiving

• Mount the LVM snapshot to a temporary or available filesystem to begin the archiving process:

$ mkdir /tmp/mongosnap
$ mount -t xfs -o nouuid,ro /dev/vg_mongo/mongosnap_20July2023 /tmp/mongosnap/
$ tar -czf /backup/mongodb_backup_$(date '+%Y%m%d%H%M').tar.gz --absolute-names /tmp/mongosnap

Explanations:

- `mkdir`: Creates a directory named `/tmp/mongosnap` to mount the LVM snapshot.

- `mount`: Mounts the LVM snapshot with read-only access using the XFS filesystem.

- `tar`: Compresses the contents of the mounted snapshot into a tar archive and saves it in the `/backup` directory. The filename includes the current date and time.

Step 6: Unlock the Database for Writing

• After the backup is complete, unlock the MongoDB database for writing using the following command:

$ mongo -u<username> -p<password> --port <port> --quiet fsyncunlock.js

cat fsyncunlock.js
--------------------------
// fsyncunlock.js
db.fsyncUnlock();

This command runs the `fsyncunlock.js` script to release the lock on the MongoDB instance, allowing write operations to resume.

Step 7: Unmount the Snapshot and Remove LVM Snap

• Finally, unmount the snapshot folder and remove the LVM snapshot to free up storage and reduce overhead:

$ umount /tmp/mongosnap
$ lvremove /dev/vg_mongo/mongosnap_20July2023

Explanations:

- `umount`: Unmounts the temporary directory where the LVM snapshot was mounted.

- `lvremove`: Removes the LVM snapshot from the system to release the allocated storage.

Automation using Bash shell script:

• This overall steps can be automated via bash shell script and can be scheduled via crontab or Job Scheduler software like Autosys.
• You can find the complete shell script by clicking here. Feel free to customize based on your need.
• As you can see from the below output the overall process took 10s since its test server not contain much data. So the overall process solely depends on database size, disk i/o and other server configurations.

[root@mongodb2 ~]# sh mongodb_snapshot_backup.sh
  Logical volume "mongosnap_20230721165554" created.
  Logical volume "mongosnap_20230721165554" successfully removed
  
[root@mongodb2 ~]# ls -lrt /backup
total 5208
-rw-r--r--. 1 root root      51 Jul 21 16:55 oplogposition_mongosnap_20230721165554.pos
-rw-r--r--. 1 root root 1771491 Jul 21 16:56 mongosnap_20230721165600.tar.gz

[root@mongodb2 ~]# tail -100 /backup_logs/backup_logs.txt
2023-07-21 16:55:54 Backup process started.
2023-07-21 16:55:54 Locking MongoDB database with fsynclock
2023-07-21 16:55:55 Recording oplog position before backup
2023-07-21 16:55:55 Creating LVM snapshot: mongosnap_20230721165554
2023-07-21 16:55:59 Mounting LVM snapshot to /tmp/mongosnap_20230721165554 with options
2023-07-21 16:56:00 LVM snapshot mounted successfully to /tmp/mongosnap_20230721165554
2023-07-21 16:56:00 Compressing LVM snapshot to /backup/mongosnap_20230721165600.tar.gz
2023-07-21 16:56:03 LVM snapshot compressed successfully to /backup/mongosnap_20230721165600.tar.gz
2023-07-21 16:56:03 Unmounting LVM snapshot from /tmp/mongosnap_20230721165554
2023-07-21 16:56:03 LVM snapshot unmounted successfully from /tmp/mongosnap_20230721165554
2023-07-21 16:56:03 Removing LVM snapshot: mongosnap_20230721165554
2023-07-21 16:56:04 Unlocking MongoDB database
2023-07-21 16:56:04 Backup process completed.
2023-07-21 16:56:04 Overall process completion time: 10 seconds

Trade-Offs of using LVM Based backups:

• Performance Impact: LVM snapshots can affect MongoDB performance during heavy write operations due to increased disk I/O.

• Consistency Concerns: LVM snapshots are crash-consistent and may lead to data inconsistencies if taken during write operations (That is the reason we are locking the database) or without including MongoDB's journal files.

• Storage Overhead: LVM snapshots consume additional disk space, potentially increasing storage costs, especially for large databases(So we need to remove the lvm snapshot as soon as backup archived )

Wrapping Up:

Implementing MongoDB backups using LVM snapshots is an efficient and reliable way to protect your critical data. By following the step-by-step process outlined in this blog post, you can confidently create consistent backups of your MongoDB database without impacting its performance. Regularly scheduled LVM snapshots combined with a well-thought-out archiving strategy will help you maintain data availability and recoverability in the face of unforeseen events.

Hope you liked the content 😄

Please provide your  valuable feedback’s in the comment section.

Keep Learning ! Keep Sharing!