PHP playing fast and loose with your data integrity

Had a potential GlusterFS user state that the filesystem incorrectly reported that a write succeeded even though all the servers were powered off. Since this sounded rather impossible, I asked for details and duplicated the problem. This is the php code:

<?php
$fp = fopen("myfootest","c");
if (fwrite($fp,"Test1\n") === false) die("Write failed.");
echo "First write completed. Turn off servers and press enter.";
fgets(STDIN);
if (fwrite($fp,"Test2\n") === false) die("Write failed.");
if (fclose($fp) !== false ) {
    echo "File closed without error.\n";
} else {
    echo "Error closing file\n";
}
?>

After opening and writing to the file once, we wait for input and pull the plugs on the servers (I actually just kill -9 the glusterfsd and glusterd processes). Then we write to the file again and close the file. We expect that either the fwrite or at least the fclose are going to error. Unfortunately, “File closed without error.”

How can this be? Can we really be accepting this write even though it can’t be written?

An strace reveals what’s actually happening though. When I strace this code, here’s what really happens:

open("/mnt/myfootest", O_WRONLY|O_CREAT, 0666) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
write(3, "Test1\n", 6)                  = 6
write(1, "First write completed. Turn off "..., 56) = 56
read(0, "\n", 8192)                     = 1
write(3, "Test2\n", 6)                  = 6
close(3)                                = -1 ENOTCONN (Transport endpoint is not connected)
write(1, "File closed without error.\n", 27) = 27

The writes don’t fail because they’re being cached. The fclose, however, is getting an error but is not passing that error along to the script. I believe that is this bug: https://bugs.php.net/bug.php?id=60110