Quick-reference procedures for operating and maintaining the live production environment. For staging operations, development workflows, and the full process library see the Staging Runbook in the staging console.
| Time On Tasks URL | https://timeontasks.laneaward.com |
|---|---|
| Console URL | https://console.laneaward.com |
| App Launchpad | https://ai.laneaward.com |
| ToT web root | /var/www/laneaward-timeontasks/ |
| Console web root | /var/www/laneaward-console/ |
| Database | /var/lib/laneaward/workforce.db |
| Backend service | laneaward-workforce-api.service · port 9194 |
| App source | /opt/laneaward/ |
| TLS cert | /etc/letsencrypt/live/timeontasks.laneaward.com/ — covers both domains, auto-renews |
| API health | https://timeontasks.laneaward.com/api/health |
| SSH | ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109 |
Every operation here targets the live production database and services. There is no undo for destructive commands. Before any operation that writes, restarts, or replaces data, ask: have I taken a snapshot and confirmed this is the right environment?
Never run database reset or activity wipe commands in production. Those procedures exist only in the staging runbook and are intentionally excluded here.
The repository at /Users/donaldscott/Project-Code/laneaward/repo/ is
under local Git version control. Deploy scripts read from the current working tree,
so the branch that is checked out at deploy time determines what gets pushed to
production. Before running any deploy, confirm the intended branch with
git branch --show-current — production deploys should generally run
from main. Full background is in the Version Control And Source
Management section of the Project Reference document.
restore_production_db.sh. Auto-selects the most recent backup or accepts a specific file. Requires two typed confirmations, takes an automatic safety backup, stops the service, restores the database, and verifies health. The service is never left stopped — recovery is attempted on failure.
Use this process when changes have been validated on staging and are ready to go live. This is the only authorized path for promoting code and data to production.
Each deploy task has a dedicated script. Using the wrong script can destroy production data.
In particular, promote_staging_db_to_production.sh replaces the entire production
database with staging — it must only be used for schema or structural database changes and
includes a mandatory double-confirmation gate. For all routine work, use the targeted scripts below.
| Task | Correct Script |
|---|---|
| Both frontends + production documents | deploy_to_production.sh |
| Time On Tasks frontend only | deploy_tot_to_production.sh |
| Operations Console frontend only | deploy_console_to_production.sh |
| Console documents only | deploy_console_docs_to_production.sh (Process 1B) |
| Backend code only (server.py or schema.sql changed) | deploy_backend_to_production.sh (Step 5) |
| ProfitMaker reference data update | push_reference_to_production.sh (Process 7) |
| Icons only | deploy_icons_to_production.sh |
| Schema or structural database change only | promote_staging_db_to_production.sh — requires double confirmation |
sqlite3 .backup — no shutdown required. Reference data (customers, orders) and all contributor activity data are never cross-promoted — each environment's operational data is always managed independently.server.py or schema.sql changed, use Step 5 (deploy_backend_to_production.sh) — not the main deploy script. It uploads backend files, restarts the service, and verifies health in one run.timeontasks/ and promotes to /var/www/laneaward-timeontasks/ops_console/ and promotes to /var/www/laneaward-console/promote_staging_db_to_production.sh for DB promotion and Process 6 for service restart.Use the Staging Runbook Process 3 (Health Check and Status) to confirm staging is healthy and the changes you intend to deploy have been tested end-to-end before proceeding.
Recommended before any deploy that includes schema changes or significant data migrations.
[VM]
sudo cp /var/lib/laneaward/workforce.db /var/lib/laneaward/workforce-pre-deploy-$(date +%F-%H%M%S).dbRun from the repo root on the Mac. The script uploads both frontends, deploys production documents, and verifies health. It does not touch the database or restart the backend.
[Mac-local]
/Users/donaldscott/Project-Code/laneaward/repo/scripts/deploy_to_production.shA clean deploy finishes with:
=== Verifying production health ===
timeontasks : {"ok": true, "status": "ok", ...}
console : {"ok": true, "status": "ok", ...}
=== Deploy complete ===
https://timeontasks.laneaward.com
https://console.laneaward.comOnly run this step if the deploy includes a schema or structural database change. This script replaces the entire production database with the staging database. It requires a manual backup, two explicit confirmations, and restarts the backend service automatically. Do not run this for frontend-only deploys, ProfitMaker updates, or any routine task.
[Mac-local]
/Users/donaldscott/Project-Code/laneaward/repo/scripts/promote_staging_db_to_production.shThe script will display both database file sizes, require you to type PROMOTE then YES, take an automatic timestamped backup, and verify health before finishing.
If backend code changed, use the dedicated backend deploy script. It uploads server.py and
schema.sql, promotes them to /opt/laneaward/workforce_app/backend/, restarts
the service, and verifies health in a single run. Schema migrations (new columns, indexes) are applied
automatically by ensure_schema_upgrades() on startup — no manual SQL required.
[Mac-local]
bash /Users/donaldscott/Project-Code/laneaward/repo/scripts/deploy_backend_to_production.shA clean run finishes with the service showing active and a live health response:
=== Production backend deploy complete ===
https://timeontasks.laneaward.com/api/healthFor routine frontend updates that do not require a DB promotion or service restart:
[Mac-local — Time On Tasks only]
/Users/donaldscott/Project-Code/laneaward/repo/scripts/deploy_tot_to_production.sh
[Mac-local — Console only]
/Users/donaldscott/Project-Code/laneaward/repo/scripts/deploy_console_to_production.sh
Two backend services run on the same VM — one for staging, one for production.
Always confirm you are targeting laneaward-workforce-api.service (production)
and not the staging service before running any systemctl command.
Use this when only documents have changed — runbook, user guide, project reference, or
environment topology. This script deploys documents only and does not touch
app.js, index.html, the database, or the backend service.
Documents deployed by this script:
_documents/production/runbook.html → runbook.html_documents/user-guide.html → user-guide.html_documents/project-reference.html → project-reference.html_documents/laneaward_environment_topology.html → topology.htmlRun from the repo root on the Mac. The script verifies all source files exist, sanity-checks that the production runbook contains no staging infrastructure references, uploads all four documents, promotes them to the production console web root, and cleans up temp files.
bash scripts/deploy_console_docs_to_production.sh
Use deploy_console_docs_to_production.sh any time only documents changed.
Use deploy_console_to_production.sh when console app code (app.js,
index.html) also changed — that script deploys both the frontend and documents
in a single run.
All LaneAward environments are protected by the 🚀 PWA BDR service — a shared macOS menu bar LaunchAgent on the development Mac. Look for the 🚀 icon in the top menu bar to open the LaneAward dropdown, trigger manual runs, or adjust schedules.
| Job | What it protects | Schedule |
|---|---|---|
VM·PROD — Database |
/var/lib/laneaward/workforce.db on AWS |
Every 12 h |
LOCAL — Source Code |
Full project on Mac, hardlink snapshots | Weekly |
~/projectbackups/laneaward/production-database/~/projectbackups/laneaward/source/~/projectbackups/backup_logs/backup.log
Each VM backup uses sqlite3 .backup — safe online copy, no downtime, no locking.
Every copy is independently restorable.
sudo cp /var/lib/laneaward/workforce.db /var/lib/laneaward/workforce-$(date +%F-%H%M%S).db
Use the dedicated restore script — see Process 2B. The script handles confirmation, safety backup, service stop/start, and health verification in a single guided run. Do not restore manually.
Use this process when production data must be recovered from a Mac-local backup. The script is fully guided — it shows you the backup and current database file sizes, requires two explicit typed confirmations, takes an automatic safety backup before making any changes, and verifies health after the restore completes.
Restoring from backup permanently destroys all production data recorded after the backup timestamp. Every work session, task, and contributor activity logged since that point will be gone. Before running this script, confirm:
RESTORE then YES — two separate gateslaneaward-workforce-api.service~/projectbackups/laneaward/production-database/workforce_vm_prod_YYYYMMDDTHHMMSS.dbls -lht ~/projectbackups/laneaward/production-database/
The restore script connects via laneaward-vm → 172.31.7.224. The Twingate client must be active before proceeding. Verify SSH access first:
[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem laneaward-vm "echo SSH OK"Run with no argument to auto-select the most recent backup, or pass a specific backup file path:
[Mac-local — most recent backup]
bash /Users/donaldscott/Project-Code/laneaward/repo/scripts/restore_production_db.sh
[Mac-local — specific backup file]
bash /Users/donaldscott/Project-Code/laneaward/repo/scripts/restore_production_db.sh ~/projectbackups/laneaward/production-database/workforce_vm_prod_TIMESTAMP.dbA clean restore finishes with:
Restore complete. Production is live and healthy.
Restored from : ~/projectbackups/laneaward/production-database/workforce_vm_prod_TIMESTAMP.db
Safety backup : /var/lib/laneaward/workforce-pre-restore-TIMESTAMP.dbConfirm production is responding correctly and the data looks right before deleting the safety backup:
[Mac-local]
curl -sS https://timeontasks.laneaward.com/api/healthOnce confirmed, delete the server-side safety backup:
[VM]
sudo rm /var/lib/laneaward/workforce-pre-restore-TIMESTAMP.dbssh laneaward-vm 'sudo systemctl status laneaward-workforce-api.service --no-pager'ssh laneaward-vm 'sudo journalctl -u laneaward-workforce-api.service -n 50 --no-pager'/var/lib/laneaward/workforce-pre-restore-TIMESTAMP.db — it can be used to roll back by running the restore script again with that file as the argument.Use these checks to confirm the production service and both apps are responding correctly.
curl -sS http://127.0.0.1:9194/api/health
curl -sS https://timeontasks.laneaward.com/api/health curl -sS https://console.laneaward.com/api/health
sudo systemctl status laneaward-workforce-api.service --no-pager
sudo journalctl -u laneaward-workforce-api.service -f
Use these only when the backend service itself needs attention. The deploy script handles restarts automatically — these are for manual intervention only.
sudo systemctl restart laneaward-workforce-api.service
sudo systemctl stop laneaward-workforce-api.service sudo systemctl start laneaward-workforce-api.service
sudo systemctl is-enabled laneaward-workforce-api.service
Two backend services run on this VM — staging and production. The production service is
laneaward-workforce-api.service. Always confirm the service name before running
any systemctl command.
Use this process when a new asidta_file_* folder arrives from ProfitMaker and you need to
refresh the production database with the latest customer numbers, customer names, order numbers, and
order descriptions.
Do not copy the local workforce.db to production. This permanently destroys users, sessions,
and activity. The process below updates only the reference tables
(customer_account, sales_order, profitmaker_import_manifest)
and never touches app_user, work_session, order_task, or any
other operational table.
scripts/refresh_workforce_reference_snapshot.sh
asidta_file_* folder if no path is passed.DBF, FPT, and CDX files into pm_database.Copy of ….workforce.db with current customers and the rolling order reference window.scripts/push_reference_to_production.sh
FORCE_REFRESH=1 to override on a weekend.
The importer recovers descriptions from AINONOTE.DBF for orders not yet promoted to
formal LNITM line items. Previously these "shell" orders displayed as
"Order NNNNNN" placeholder text in Time On Tasks and the Operations Console. The first
push after the AINONOTE fallback was deployed updates approximately 50 existing placeholders to
real product descriptions, and the daily push continues to catch new shell orders on every
subsequent run.
"plaque polshd blck acrylc 9" x 12"".AINONOTE.ONLINENO position,
for example "(1) Flag case walnut to hold 3' x 5' flag (2) Instal of AZ Flags in flags
are Cust Provided"."Order NNNNNN" — correct
behavior when ProfitMaker has nothing more.
No operational action required. Schema is unchanged; this is purely an enrichment of the
sales_order.description field. Implementation lives in
workforce_app/backend/import_profitmaker_reference.py
(build_item_description_lookup and supplement_summaries_from_ainonote).
/Users/donaldscott/Project-Code/laneaward/repo/scripts/refresh_workforce_reference_snapshot.sh
This script is fully idempotent — it is safe to run more than once against the same
asidta_file_* snapshot. The importer uses upserts throughout, so re-running it will
simply overwrite the reference tables with identical data and leave net zero change. No records are
deleted, no operational tables are touched, and no counters are incremented. If you are unsure
whether Step 1 has already run for the current snapshot, run it again — it will not cause any harm.
Before pushing to production, confirm that staging has already received the same reference update. Follow Staging Runbook Process 7 (Update SQLite From ProfitMaker Files) to push to staging and verify it is healthy before proceeding here.
Run independently against production. Updates only the reference tables in the production database — never touches contributor sessions, tasks, users, or any operational data.
/Users/donaldscott/Project-Code/laneaward/repo/scripts/push_reference_to_production.sh
If today is a weekend the script will exit early with a reminder. Override with:
FORCE_REFRESH=1 bash /Users/donaldscott/Project-Code/laneaward/repo/scripts/push_reference_to_production.sh
curl -sS https://staging.timeontasks.laneaward.com/api/health curl -sS https://timeontasks.laneaward.com/api/health
Both should return {"ok": true, ...}. Then confirm order data is searchable on both environments:
curl -sS "https://staging.timeontasks.laneaward.com/api/orders/search?q=107923&limit=3" curl -sS "https://timeontasks.laneaward.com/api/orders/search?q=107923&limit=3"
Both should return a matching order row with data_source: "PROFITMAKER".
Phase 1 durability hardening is part of the production backend and should be preserved whenever the
application programming interface, or API, is updated. The Time On Tasks API opens SQLite in
Write-Ahead Logging (WAL) mode, waits up to 10 seconds for short lock contention, uses
synchronous = FULL for safer commits, and wraps each mutating route in a short
BEGIN IMMEDIATE write transaction.
PRAGMA journal_mode = WAL — Write-Ahead LoggingPRAGMA busy_timeout = 10000 — up to 10,000 ms wait on a short lockPRAGMA synchronous = FULL — favors safer disk writes over speedHTTP 503 Service Unavailableretryable: trueIteration test checklist:
Five database-reliability features are fully implemented and should remain in place together:
Write-Ahead Logging (WAL)busy_timeout = 10000synchronous = FULLBEGIN IMMEDIATE write transactionsHTTP 503 responses for SQLite busy/locked contentionPhase 2 (client-side retry/backoff, temporary local storage, idempotent write keys) was evaluated and deferred. A concurrent stress test at 2× the expected user load passed cleanly with significant headroom — Phase 1 alone is sufficient at current scale. Reconsider only if load grows significantly.
Time On Tasks includes a service worker at
timeontasks/sw.js
that improves load speed on shared tablets by caching static assets locally after the first visit.
Understanding the cache strategy is important before deploying any frontend changes.
| Request type | Strategy | Why |
|---|---|---|
HTML documents (index.html, user-guide.html) | Network-first | Always fetches fresh HTML so deployed updates are visible on next page load without any SW changes. |
Versioned static assets (app.js?v=…, icons, manifest) | Cache-first | Version token in the URL acts as the cache key. New token = new URL = automatic cache miss = fresh fetch. |
/api/* and all non-GET requests | Network-only | Task writes, session state, and PIN login must never be served from cache. |
No changes to sw.js are required. The version token does the work.
app.js or other assets.index.html (e.g. app.js?v=20260406-foreman1 → app.js?v=20260411-myfix1).index.html and the updated asset file via Process 1.index.html (network-first), browser sees the new token URL, cache misses, fetches new asset, caches it. Done.CACHE_VERSION in sw.js. The new SW deletes all prior caches on activate.When updating sw.js, the browser detects the change automatically (byte-for-byte comparison on every page load). The new SW installs in the background, then activates and claims all open tabs immediately via skipWaiting and clients.claim.
index.html. The old token URL stays in cache and will be served.app.js as transferred, or add --checksum to force a content comparison.index.html to a cache-first rule. HTML must always be network-first or the stale-app-shell problem returns.
Run this test against production to confirm that the Phase 1 reliability improvements (WAL,
busy_timeout, synchronous = FULL, BEGIN IMMEDIATE) hold up
under the expected concurrent load of up to 20 simultaneous tablet operators on the shop floor.
This test was completed before go-live and passed cleanly. The procedure is preserved here as a
reference for future validation runs (e.g. after significant backend changes or scale increases).
The test script is at workforce_app/backend/stress_test_concurrent.py and runs from
your local Mac. It seeds its own test fixtures into the production database via SSH, runs the load,
then cleans up after itself.
ubuntu@3.130.69.109 with no passphrase prompt (BatchMode).https://timeontasks.laneaward.com.cd /Users/donaldscott/Project-Code/laneaward/repossh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109 echo "SSH OK"Simulates 20 human-paced operators for 2 task cycles each. Expected wall time is roughly 20–40 seconds.
python3 workforce_app/backend/stress_test_concurrent.py --host https://timeontasks.laneaward.comA clean pass looks like this:
====================================================================
LANEAWARD TIME-ON-TASKS — CONCURRENT STRESS TEST
====================================================================
Mode: REALISTIC (human-paced 1.5–4.0 s)
Target: https://timeontasks.laneaward.com
Concurrent users: 20
Iterations/user: 2
Total ops: 280
Wall time: 31.4s
Outcomes:
Successes: 280 (100%)
Hard failures: 0 (non-retryable errors or timeouts)
Retryable busy: 0 (SQLite busy-wait — server queued OK)
VERDICT
------------------------------------------------------------------
PASS All operations completed cleanly under concurrent load.python3 workforce_app/backend/stress_test_concurrent.py --host https://timeontasks.laneaward.com --burstExpect more retryable busy responses in this mode — that is normal. Zero hard failures is still required.
--users N — number of concurrent users, 1–20 (default: 20)--iterations N — task cycles per user (default: 2)--burst — near-simultaneous writes, ceiling test only--no-seed — skip seeding (test users already in DB from prior run)--no-cleanup — leave test fixtures in DB for inspectionsudo journalctl -u laneaward-workforce-api.service -n 100ping timeontasks.laneaward.combash scripts/cleanup_stress_test_production.sh
The server is protected by two layers: an AWS Security Group that restricts SSH (port 22) to authorized IP addresses, and a Twingate connector that allows SSH from any location through the Twingate client. Use this process to connect to the server and to update security group rules when IP addresses change.
| Group ID | sg-0cc9719fa0e029c40 (launch-wizard-1) |
|---|---|
| COX Fiber — office | 98.175.1.150/32 · SSH allowed |
| COX Cable failover — office | 72.215.199.214/32 · SSH allowed |
| Home lab (pending removal) | 72.208.129.218/32 · SSH allowed |
| HTTP — port 80 | Open to all — 0.0.0.0/0 · required for Let's Encrypt HTTP-01 renewal |
| HTTPS — port 443 | Open to all — 0.0.0.0/0 |
| Remote Network | Lane Award PWA Server |
|---|---|
| Connector | eggplant-okapi |
| Resource address | 172.31.7.224 (server private IP) |
| SSH key | ~/.ssh/lane_webserver.pem |
Inbound port 80 (0.0.0.0/0) must stay open. Let's Encrypt renews every
certificate via the HTTP-01 challenge, which the Certificate Authority validates over
port 80. nginx redirects all real traffic from 80 to 443, so the only
thing port 80 serves is the ACME challenge — but if it is closed, every renewal
fails silently: HTTPS keeps working on the existing certs while they quietly march toward
expiry. This exact failure was found and fixed on 2026-06-29 (port 80 had been removed,
and two certs had already lapsed). If a renewal ever fails, first confirm port 80 is open:
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions[?FromPort==\`80\`]" --no-cli-pager.
Re-add with:
aws ec2 authorize-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 80 --cidr 0.0.0.0/0.
Use this method when connecting from home or any location not on an authorized static IP. The Twingate client must be running and connected before opening SSH.
The Twingate icon lives in the Mac menu bar. Click it and verify the connection status is active.
[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@172.31.7.224Use this method when connecting from the office on either the fiber or cable connection. Twingate does not need to be running.
[Mac-local]
ssh -i ~/.ssh/lane_webserver.pem ubuntu@3.130.69.109
When the Twingate client is active, it intercepts connections to the server's public IP and
routes them through the connector. Pause Twingate first before using Option B, or use the
private IP (172.31.7.224) with Twingate active instead.
Run this on the server to confirm the connector service is running. A healthy connector shows State: Online in the log output.
[VM]
sudo systemctl status twingate-connector --no-pagerRun these steps when an authorized IP address changes. Requires the AWS CLI configured on the development Mac with IAM user donald.
[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pager[Mac-local]
aws ec2 revoke-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr OLD.IP.ADDRESS/32[Mac-local]
aws ec2 authorize-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr NEW.IP.ADDRESS/32[Mac-local]
aws ec2 describe-security-groups --group-ids sg-0cc9719fa0e029c40 --query "SecurityGroups[0].IpPermissions" --output json --no-cli-pagerDo not remove an IP that is your current connection without first confirming Twingate SSH works, or without another authorized IP still in place. If all SSH access is lost, recovery requires the AWS Console. Never remove all three SSH rules at once.
Once Twingate is confirmed as the primary home access method, run this to remove the dynamic home lab IP. Do not run this until Twingate SSH has been verified working from the home location.
[Mac-local]
aws ec2 revoke-security-group-ingress --group-id sg-0cc9719fa0e029c40 --protocol tcp --port 22 --cidr 72.208.129.218/32
Use this to confirm all active user PINs have been migrated to Argon2id hashes and that the fast-login
HMAC token has been populated. Both columns must be set for a user to take the fast login path (~230 ms).
Users with an empty pin_token will take the slow fallback path on their next login, which
automatically writes the token — no manual action required.
sqlite3 'file:/var/lib/laneaward/workforce.db?immutable=1' 'SELECT id, display_name, CASE WHEN pin_code LIKE "$argon2%" THEN "hashed" ELSE "plaintext" END AS hash_status, CASE WHEN pin_token != "" THEN "token-ok" ELSE "no-token" END AS token_status FROM app_user WHERE is_active = 1 ORDER BY display_name;'
Every active user should show hashed and token-ok. If any show
plaintext or no-token, that user will be upgraded automatically on their next
successful login — no manual action required. The slow fallback path handles both cases gracefully.
Run this on the Mac to compute the expected HMAC token for a known PIN and compare against the database:
python3 -c "import hmac; print(hmac.new(b'laneaward-pin-pepper-v1', b'444444', 'sha256').hexdigest())"
Compare the output against the pin_token column for the user with that PIN. They must match exactly.
Salaried contributors log time through the Time On Tasks app exactly like hourly workers. The Order Profitability report converts their annual salary to an effective hourly rate using the U.S. Bureau of Labor Statistics standard:
Effective Hourly Rate = Annual Salary ÷ 2,080
Session Labor Cost = (Annual Salary ÷ 2,080) × (Session Minutes ÷ 60)
2,080 = 52 weeks × 40 hours — the standard used by ADP, Paychex, QuickBooks, and the BLS.