Okay, so storage wasn't quite ready to go.
We had been a bit too trusting and not properly vetted certain whispered rumors of the dark and horrible consequences of letting your LVM snapshots fill up. In our defense, the way that people generally use LVM snapshots makes it very unlikely that they'll fill, and apparently it's not a commonly-seen failure, disk being cheap. . .
Anyway. No excuse. Experimental verification is the cornerstone of science, so we tested it.
# lvcreate -n origin -L 1G LogVol01
# lvcreate -n snap -L 100M --snapshot LogVol01/origin
Now, once we've made 100M of changes to origin, testsnap should fill up. If you've been reading the LVM snapshot warnings, the earth will then erupt in fire, pitch will rain from the sky, the crust will split and a cavernous maw with teeth the size of the Tokyo Tower will emerge to consume humanity. The lucky portion of it, anyway.
This did not happen. We filled it by making a filesystem and copying stuff in from /usr . Turns out that the machine keeps running merrily. There are still errors, of course:
device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception
Followed by a bunch of errors of the form:
Buffered I/O error on device dm-3, logical block 585
We unmounted the snapshot, tried to remount it, no dice. It was, as McCoy would say, dead. The original LV was fine, however.
We had been a bit too trusting and not properly vetted certain whispered rumors of the dark and horrible consequences of letting your LVM snapshots fill up. In our defense, the way that people generally use LVM snapshots makes it very unlikely that they'll fill, and apparently it's not a commonly-seen failure, disk being cheap. . .
Anyway. No excuse. Experimental verification is the cornerstone of science, so we tested it.
# lvcreate -n origin -L 1G LogVol01
# lvcreate -n snap -L 100M --snapshot LogVol01/origin
Now, once we've made 100M of changes to origin, testsnap should fill up. If you've been reading the LVM snapshot warnings, the earth will then erupt in fire, pitch will rain from the sky, the crust will split and a cavernous maw with teeth the size of the Tokyo Tower will emerge to consume humanity. The lucky portion of it, anyway.
This did not happen. We filled it by making a filesystem and copying stuff in from /usr . Turns out that the machine keeps running merrily. There are still errors, of course:
device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception
Followed by a bunch of errors of the form:
Buffered I/O error on device dm-3, logical block 585
We unmounted the snapshot, tried to remount it, no dice. It was, as McCoy would say, dead. The original LV was fine, however.
Leave a comment