sorry for the downtime... the filesystem filled up which took the site down
we're back for right now, although we still need to brew some coffee and fix the root cause
@Accalia so here's what happened
the hourly db backup caused the disk to hit 85% usage
this caused elasticsearch to trip out
this caused all of the ES jobs to fail
each failing job produced a traceback
these tracebacks went into the docker console log for that container
the logs filled up the disk
the disk filling up crashed everything
and i had to get out of bed to fix it
@rey ...... oofh. cascading failures.....
well at least that's sorted out now, right?
@Accalia i hope so!!
The Vulpine Club is a friendly and welcoming community of foxes and their associates, friends, and fans! =^^=