Time On Tasks & Operations Runbook

Process 2: Backup and Recovery

Production is protected by three independent layers. They are deliberately different from one another: each one survives a failure that would defeat the others.

Layer 1 — Offsite Copies To Amazon S3

The server backs itself up and pushes the result outward. The Mac plays no part, so this layer keeps working whether or not the Mac is switched on.

systemctl list-timers laneaward-s3-backup.timer --no-pager
sudo tail -20 /var/log/laneaward-s3-backup.log
sudo journalctl -u laneaward-s3-backup.service -n 20 --no-pager

sudo systemctl start laneaward-s3-backup.service

Layer 2 — Local Copies On The Development Mac

The PWA BDR service — a menu bar application on the Mac, shown as a 🚀 icon — pulls copies down every twelve hours. Open the icon to see status, trigger a manual run, or adjust schedules. This layer is what the guided restore procedure in Process 2B uses, because a local file restores faster than anything fetched over the internet.

grep "LANEAWARD/VM" ~/projectbackups/backup_logs/backup.log | tail -6

Layer 3 — Machine Images

The database layers protect your data. They do not protect the machine — the operating system, installed packages, service definitions, web server configuration and certificates. A machine image captures all of that, so the server can be rebuilt rather than reconstructed from memory. See Process 2C for how to use one.

sudo cp /var/lib/laneaward/workforce.db /var/lib/laneaward/workforce-$(date +%F-%H%M%S).db

Process 3: Health Check and Status

Use these checks to confirm the production service and both apps are responding correctly.

curl -sS http://127.0.0.1:9194/api/health

curl -sS https://timeontasks.laneaward.com/api/health
curl -sS https://console.laneaward.com/api/health

sudo systemctl status laneaward-workforce-api.service --no-pager

sudo journalctl -u laneaward-workforce-api.service -f

Process 8: Reliability Verification For Task Writes

Phase 1 durability hardening is part of the production backend and should be preserved whenever the application programming interface, or API, is updated. The Time On Tasks API opens SQLite in Write-Ahead Logging (WAL) mode, waits up to 10 seconds for short lock contention, uses synchronous = FULL for safer commits, and wraps each mutating route in a short BEGIN IMMEDIATE write transaction.

Iteration test checklist:

• Open two or more contributor-capable sessions on production.
• Rapidly add, start, pause, complete, and cancel tasks from separate browsers or tablets.
• Confirm each UI action returns to a stable task state without duplicate sessions or duplicate task inserts.
• If a write is delayed by contention, confirm the API reports a retryable busy condition instead of a generic server error.
• After the test pass, verify the affected task lists and completed records still match the contributors who performed the actions.

Process 9: Service Worker — Deploy Model And Cache Invalidation

Time On Tasks includes a service worker at timeontasks/sw.js that improves load speed on shared tablets by caching static assets locally after the first visit. Understanding the cache strategy is important before deploying any frontend changes.

Process 10: Concurrent Stress Test

Run this test against production to confirm that the Phase 1 reliability improvements (WAL, busy_timeout, synchronous = FULL, BEGIN IMMEDIATE) hold up under the expected concurrent load of up to 20 simultaneous tablet operators on the shop floor. This test was completed before go-live and passed cleanly. The procedure is preserved here as a reference for future validation runs (e.g. after significant backend changes or scale increases).

The test script is at workforce_app/backend/stress_test_concurrent.py and runs from your local Mac. It seeds its own test fixtures into the production database via SSH, runs the load, then cleans up after itself.

Prerequisites

• SSH key access to ubuntu@3.130.69.109 with no passphrase prompt (BatchMode).
• Production server running and reachable at https://timeontasks.laneaward.com.
• Python 3.10 or later on your local Mac — no third-party packages required.
• Run from the repo root: cd /Users/donaldscott/Project-Code/laneaward/repo

Step 1 — Verify SSH access before starting

ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109 echo "SSH OK"

Step 2 — Run the standard realistic test

Simulates 20 human-paced operators for 2 task cycles each. Expected wall time is roughly 20–40 seconds.

python3 workforce_app/backend/stress_test_concurrent.py --host https://timeontasks.laneaward.com

A clean pass looks like this:

====================================================================
  LANEAWARD TIME-ON-TASKS  —  CONCURRENT STRESS TEST
====================================================================
  Mode:             REALISTIC (human-paced 1.5–4.0 s)
  Target:           https://timeontasks.laneaward.com
  Concurrent users: 20
  Iterations/user:  2
  Total ops:        280
  Wall time:        31.4s

  Outcomes:
    Successes:        280  (100%)
    Hard failures:      0  (non-retryable errors or timeouts)
    Retryable busy:     0  (SQLite busy-wait — server queued OK)

  VERDICT
  ------------------------------------------------------------------
  PASS  All operations completed cleanly under concurrent load.

Step 3 — Optional: run the burst ceiling test

python3 workforce_app/backend/stress_test_concurrent.py --host https://timeontasks.laneaward.com --burst

Expect more retryable busy responses in this mode — that is normal. Zero hard failures is still required.

Common Options

If the test fails

bash scripts/cleanup_stress_test_production.sh

Process 11: Server Access And Security Group Management

The server is protected by two layers: an AWS Security Group that restricts SSH (port 22) to authorized IP addresses, and a Twingate connector that allows SSH from any location through the Twingate client. Use this process to connect to the server and to update security group rules when IP addresses change.

Host laneaward-vm 3.130.69.109
  HostName 3.130.69.109
  User ubuntu
  IdentityFile /Users/donaldscott/.ssh/lane_webserver.pem
  IdentitiesOnly yes

Option A — SSH via Twingate (from any location)

Use this method when connecting from home or any location not on an authorized static IP. The Twingate client must be running and connected before opening SSH.

Step 1 — Open the Twingate client on your Mac and confirm it shows Connected

The Twingate icon lives in the Mac menu bar. Click it and verify the connection status is active.

Step 2 — SSH using the server private IP

[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@172.31.7.224

Option B — SSH directly from an authorized office IP

Use this method when connecting from the office on either the fiber or cable connection. Twingate does not need to be running.

Step 1 — SSH using the server public IP

[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109

Check Twingate Connector Status

Run this on the server to confirm the connector service is running. A healthy connector shows State: Online in the log output.

[VM]
sudo systemctl status twingate-connector --no-pager

Update Security Group — Replace a Changed IP

Run these steps when an authorized IP address changes. Requires the AWS CLI configured on the development Mac with IAM user donald.

Step 1 — Verify current rules before making changes

[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pager

Step 2 — Remove the old IP

[Mac-local]
aws ec2 revoke-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr OLD.IP.ADDRESS/32

Step 3 — Add the new IP

[Mac-local]
aws ec2 authorize-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr NEW.IP.ADDRESS/32

Step 4 — Verify the updated rules

[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pager

Authorized SSH Sources — What Each Rule Is For

Security group sg-0cc9719fa0e029c40 permits SSH on port 22 from three addresses. Before changing or removing any of them, understand what depends on it.

ssh laneaward-vm 'echo ${SSH_CLIENT%% *}'

Login Session Behavior

Signing in with a PIN issues a login session that lasts 12 hours as a sliding window — each use resets the clock, so an actively used device stays signed in and only idle time expires it. After roughly 12 hours of inactivity the app returns to the sign-in screen; the contributor or manager simply re-enters their PIN. No data is lost.

An automatic logout does not stop a running task timer. Work sessions live in the database and keep running until someone uses Pause, Stop, or Complete, or a manager applies a correction in the console. If a tablet logs itself out overnight while a task was left active, that task is still running — treat it as a potential runaway timer and correct it; do not assume the logout closed it.

Salary Labor Cost — Calculation Methodology

Salaried contributors log time through the Time On Tasks app exactly like hourly workers. The Order Profitability report converts their annual salary to an effective hourly rate using the U.S. Bureau of Labor Statistics standard: