Follow
sorry for the downtime... the filesystem filled up which took the site down
we're back for right now, although we still need to brew some coffee and fix the root cause
@Accalia so here's what happened
the hourly db backup caused the disk to hit 85% usage
this caused elasticsearch to trip out
this caused all of the ES jobs to fail
each failing job produced a traceback
these tracebacks went into the docker console log for that container
the logs filled up the disk
the disk filling up crashed everything
and i had to get out of bed to fix it
@rey ...... oofh. cascading failures.....
"fun"
well at least that's sorted out now, right?
..... right?
...
... RIGHT?!
...
🤣
@rey I had wondered what happened....
Log files failed to rotate and blew up your storage, or the slightly harder solution of "people keep posting too many cute fox videos and pictures they're taking all my storage!"
wait.... it's gotta be the former, there's no such thing as too many cute fox videos and pictures. :-D