Follow

sorry for the downtime... the filesystem filled up which took the site down

we're back for right now, although we still need to brew some coffee and fix the root cause

Β· Β· 2 Β· 4 Β· 11

@rey I had wondered what happened....

Log files failed to rotate and blew up your storage, or the slightly harder solution of "people keep posting too many cute fox videos and pictures they're taking all my storage!"

wait.... it's gotta be the former, there's no such thing as too many cute fox videos and pictures. :-D

@Accalia so here's what happened

the hourly db backup caused the disk to hit 85% usage

this caused elasticsearch to trip out

this caused all of the ES jobs to fail

each failing job produced a traceback

these tracebacks went into the docker console log for that container

the logs filled up the disk

the disk filling up crashed everything

and i had to get out of bed to fix it

@rey ...... oofh. cascading failures.....

"fun"

well at least that's sorted out now, right?

..... right?

...

... RIGHT?!

...

🀣

Sign in to participate in the conversation
The Vulpine Club

The Vulpine Club is a friendly and welcoming community of foxes and their associates, friends, and fans! =^^=