New Milleotto /disk/global hang
/disk/global on Milleotto has been hung again and releasing it caused the loss of many jobs.
Despite our best efforts to prevent /disk/global from hanging, it did happen again. The cause was a time bomb. A special server did not get its clock updated as it should and had slowly inched an hour ahead, which caused the hang when it tried to write to the filesystem. The clock has been reset, so this particular problem should not occur again any time soon.
Unfortunately, the release of a total hang means that many jobs may crash. Even jobs that have been queueing may be affected, as we just discovered. This is because there apparently is a time lag, between the release of a hanging job that gets killed and the return of the node to normal, meaning that jobs entering the node may also die.
We are really sorry for the inconvenience.
