DISASTER RECOVERY

Backup & Restore Operations

VIRCON implements automated database backup workflows with encrypted storage, integrity verification, and disaster recovery procedures. Backups ensure data safety during migrations, policy changes, and unexpected failures.

Backup Commands

Commands for creating, verifying, and restoring database backups.

,backup create [label] - Manually trigger database backup
,backup list [days] - View all available backups
,backup info <backup_id> - Show backup details and verification status
,backup restore <backup_id> - Restore database from backup (requires confirmation)
,backup verify <backup_id> - Test backup integrity without restoring
,backup export <backup_id> - Download backup file for offline storage

Backup Types

Automated and manual backup categories with retention policies and scheduling.

Auto-Daily

Frequency: Daily at 03:00 UTC

Retention: 30 days rolling window

Automated full database backup for routine recovery

Auto-Weekly

Frequency: Sundays at 04:00 UTC

Retention: 12 weeks (3 months)

Weekly snapshots for longer-term recovery needs

Auto-Monthly

Frequency: 1st of month at 05:00 UTC

Retention: 12 months (1 year)

Long-term archival backups for historical reference

Pre-Deployment

Frequency: Before bot restarts/migrations

Retention: Until deployment verified successful (7 days minimum)

Safety net before risky operations

Manual

Frequency: On-demand via ,backup create

Retention: 90 days unless labeled for longer retention

User-initiated backups before policy changes or testing

Backup Creation Workflow

Automated process for encrypting, compressing, and verifying database backups.

1. Backup triggered automatically by schedule or manually by admin.
2. Bot places database in read-only mode momentarily (sub-second lock).
3. Full database exported to compressed SQLite backup file.
4. Backup file encrypted with AES-256 for security.
5. Integrity hash (SHA-256) calculated and stored with backup metadata.
6. Backup uploaded to primary storage (local) and remote backup storage (cloud).
7. Verification test run to ensure backup can be read and parsed.
8. Backup entry logged to audit trail with size, duration, and status.
9. Success/failure notification sent to admin channel.

Restore Workflow

Critical step-by-step recovery process with safety confirmations and verification.

1. Admin identifies backup to restore using ,backup list.
2. Admin runs ,backup restore <backup_id> command.
3. System prompts for confirmation with warning (destructive operation).
4. Admin confirms with secondary command or reaction.
5. Bot announces maintenance mode to all staff (no actions permitted).
6. Current database moved to timestamped rollback file (safety copy).
7. Backup file downloaded, decrypted, and integrity verified.
8. Database restored from backup file, indexes rebuilt.
9. Bot restarts with restored database, runs integrity checks.
10. Admin reviews restore success, confirms data accuracy.
11. Bot exits maintenance mode, normal operations resume.

Backup Metadata Schema

Database structure for tracking backup files, integrity hashes, and retention schedules.

Field	Description
backup_id	Unique backup identifier.
backup_timestamp	When backup was created (UTC).
backup_type	auto_daily, auto_weekly, auto_monthly, pre_deploy, manual.
backup_label	Optional human-readable label for manual backups.
created_by	Discord ID of admin (NULL for automated backups).
file_path	Local storage path to encrypted backup file.
remote_path	Cloud storage URL for redundancy.
file_size_bytes	Backup file size (compressed and encrypted).
integrity_hash	SHA-256 hash of backup file contents.
verified_at	Last successful verification timestamp.
retention_until	Scheduled deletion date per retention policy.

Backup Verification Tests

Automated integrity checks ensuring backups are valid and restorable.

File integrity: SHA-256 hash matches recorded value.
Decryption: backup file can be decrypted without errors.
Schema validation: database structure matches expected schema.
Row count: sanity check that table row counts are reasonable.
Foreign key constraints: all relationships intact (no orphaned rows).
Critical data: verify key tables (users, moderation, config) are present.

Disaster Recovery Scenarios

Common failure scenarios with detection methods and recovery procedures.

Database Corruption

Detection: Bot crashes on startup, reports database integrity errors

Recovery: Restore from most recent verified auto-daily backup (max 24h data loss)

Time: 5-10 minutes for restore + verification

Accidental Mass Ban

Detection: Admin mistakenly runs bulk ban on wrong user list

Recovery: Restore from pre-action manual backup created 10 minutes prior

Time: 5 minutes restore, no data loss

Failed Bot Migration

Detection: New bot version incompatible with database schema

Recovery: Restore from pre-deployment backup, rollback to previous bot version

Time: 10-15 minutes to restore and redeploy old bot

Hosting Provider Outage

Detection: Primary server unresponsive for >30 minutes

Recovery: Failover to backup server, restore from remote cloud backup

Time: 15-30 minutes to spin up new instance and restore

Automation Features

Scheduled backups, cloud synchronization, and automatic retention management.

Scheduled backups: daily/weekly/monthly auto-execution with no admin intervention.
Pre-deployment hooks: automatic backup before bot restarts or migrations.
Backup verification: automated integrity testing after each backup creation.
Cloud sync: automatic upload to remote storage (Amazon S3 or equivalent).
Retention pruning: old backups auto-deleted per retention policy to save storage.
Health monitoring: alerts if backup fails or verification finds corruption.

Best Practices

Critical guidelines for backup strategy, testing, and disaster preparedness.

ALWAYS create manual backup before policy changes or risky operations.
Test restore procedure quarterly to ensure backups actually work.
Verify backup integrity monthly using ,backup verify command.
Keep at least 3 backup copies: local, remote cloud, offline archive.
Document restore procedure in operations runbook for emergency reference.
Monitor backup disk space to prevent failures due to storage exhaustion.

Limitations & Caveats

Important scope limitations and performance considerations for backups.

Backups do NOT include Discord message history (only metadata like mod actions).
Backups do NOT include VRChat API data (only VRChat IDs synced to bot).
Restore operation causes 5-10 minute downtime (bot unavailable during restore).
Large databases (>500MB) take longer to backup and restore.
Cloud backup sync depends on internet connectivity (local backup always succeeds).