Ahead of running this against a live site (not just staging bulk-imports),
addressed the two risks that only matter under concurrent traffic:
- One bad product no longer wastes a whole batch: each product save is
wrapped in its own SAVEPOINT within the batch transaction, so a failure
rolls back just that row (counted in the new failed[] result, surfaced
via WP_CLI::warning like import-gutenberg already does) instead of
discarding up to 200 already-good rows. Verified SAVEPOINT/ROLLBACK TO
SAVEPOINT actually works on this MariaDB instance via a direct $wpdb test
before relying on it.
- Cache invalidation suspension is now scoped per-batch instead of the
whole run, with clean_post_cache() called explicitly per product once
each batch commits, replacing the single end-of-run wp_cache_flush().
This site runs a shared Redis object cache across PHP-FPM workers, so a
whole-run suspension + full flush would let a concurrent visitor see
stale cached product data for the run's duration, and the flush itself
would evict sessions and everything else site-wide. Scoping bounds
staleness to one batch and targets only the products actually touched.
Also moved term resolution (term_exists()/wp_insert_term()) out of the
per-product savepoint entirely: warm_term_cache() now resolves every
distinct subject for a batch before its transaction opens, so a term
created for one product can never get silently rolled back by a *different*
product's savepoint failure while a stale term_id sits cached in memory
and gets attached to a later product (an orphaned term_relationships row,
checked for directly post-fix: 0 found).
Verified after: 2000-item sync still ~90s (savepoints add no measurable
overhead), 0 lookup-table mismatches, 0 term-count mismatches, 0 orphaned
term_relationships, idempotent re-run with 0 duplicates.
Measured 2000-product sync_all() at ~2.5 products/sec baseline (2-core/8GB
staging box). Confirmed via source inspection that WooCommerce's
wc_product_meta_lookup sync is synchronous inside WC_Product::save(), so
every product save was its own implicit autocommit transaction -> its own
fsync. Batching each 200-item chunk into one explicit transaction, deferring
product_cat term counting, suspending post cache invalidation for the
duration, and caching subject->term_id lookups in memory brings the same
2000-item sync to ~21 products/sec (~95s), with the update-path re-run at
~82/sec.
Verified correctness after, not just speed: wc_product_meta_lookup price/
stock matched postmeta exactly (0 mismatches across all 2000), product_cat
term counts matched actual term_relationships (0 mismatches), no orphaned
category assignments, no duplicate products on re-run.
Cover sync (OpenLibraryAdapter + CoverSync + sync-covers command): fetches
a real cover from Open Library's free, keyless, explicitly-licensed-for-
this-use Covers API and attaches it as a genuine Media Library attachment
(not a hotlinked <img> — WooCommerce's shop loop/gallery/structured data
all need a real _thumbnail_id). ISBN-first, title/author-search fallback
via the confirmed cover_i field, matching the pattern already established
in HardcoverAdapter.
Two real bugs found and fixed while verifying this against the actual
catalog, not assumed:
- A Range-header HEAD-equivalent probe (added to avoid double-fetching)
caused Open Library's server to redirect with a misleading content-type,
producing a false positive on a known-fake ISBN. Removed — fetch once,
verify the real bytes.
- Their search endpoint has genuine transient failures under repeated
querying (same request, same input, failed then succeeded seconds
later) — added retry-with-backoff on network errors/5xx, matching how
the rest of this codebase already treats transient failures as
retriable rather than fatal.
Also found chasing what looked like a third cover-sync bug, but wasn't
one: uploads/2026/08 was owned by root, silently blocking www-data-run
wp-cli from writing new files. Root cause was a gap in the earlier
root-hardening pass (docker-compose.yml, Commands.php) — it fixed our own
deploy.sh/backup.sh/Makefile invocations but missed the cron sidecar's
own internal process, which was still running its wp-cli loop as root via
a leftover --allow-root. Fixed at the container level (`user: www-data`
on the cron service) since it's a plain shell loop with none of
php-fpm's master-process-needs-root-to-drop-privileges concern.
Gutenberg importer: filters to actual literature via LoCC (Library of
Congress Classification) — verified directly against the real catalog
that novels consistently get a P* code while government documents/
speeches/law get E/JK/KF/DA and never a P code. Removes the Declaration
of Independence, Bill of Rights, etc. from what was importing as "books."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four flat taxonomies attached to product (bsc_genre, bsc_mood,
bsc_content_warning, bsc_tag) reuse WordPress's native taxonomy archive
system for the "click a tag, see everything with it" browsing AO3 is known
for — no custom archive templates or query logic needed. Verified: all
four register correctly, render as clickable chips on the product page
(content warnings get a distinct notice instead of just another chip),
and the archive page + term count both work end-to-end with real test
data.
HardcoverAdapter + `wp bookstore sync-hardcover-tags` pull genre/mood/
content-warning/freeform tags from Hardcover's API to populate these.
This part is explicitly NOT verified against a live response — Hardcover's
API is in beta with informal docs, and there was no API token available
to confirm the exact query/response shape. Flagged clearly in the adapter
itself; needs a real token + introspection query before trusting the
field-parsing logic in production. ISBN-first-then-title/author matching
handles both real future ISBNs and the current Gutenberg-synthetic
catalog's fake ones.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--allow-root was routing around wp-cli's own safety check rather than
addressing why root was there in the first place: docker exec defaults to
the container's root user because the image never sets a non-root user
for exec sessions — it was never about the actual web-facing attack
surface, which already runs as www-data (verified: php-fpm's worker
processes, the ones executing plugin/theme code for real requests, run as
uid 33, not root; only our own deliberate admin commands were root).
wp-config.php and the rest of wp-core are already owned by www-data (the
official entrypoint sets this up), so there's no actual reason for our
own commands to run as anything else. Verified: a full clean deploy and
a bookstore-core wp-cli command both work identically running as
www-data, no functional change, just removed an unnecessary privilege.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds bookstore-core's first real business logic (design doc §02-§05,§07),
replacing the plugin stub with:
- Schema: bsc_work/edition/isbn/cover_variant/supplier_offer/member tables
via dbDelta, with a versioned upgrade path.
- GutenbergImporter: imports Project Gutenberg's live bulk catalog as
synthetic staging data. Deterministic checksum-valid ISBN-13s (there are
no real ISBNs in PG data) keep re-imports idempotent.
- SyntheticOfferGenerator: the GutenbergTestAdapter of §04 — realistic
condition/stock/price distribution with deliberate out-of-stock/stale/
discontinued cases for later exception-path testing.
- PricingEngine: the §05 margin formula, verified against the design
doc's own worked example ($6.00 + $3.99 -> $14.99 exactly).
- ProductSync: one WooCommerce product per Work, never per ISBN (§03),
idempotent (verified: re-sync of 2000 works produces 0 duplicates).
- BSC_Free_Shipping: a real WC_Shipping_Method (WooCommerce's built-in
Free Shipping only supports one static threshold, not membership-
conditional), verified through actual browser-session cart flows in
both directions, not just unit calls.
Found and fixed one real data-integrity bug while scale-testing: PG's
Authors field lists every contributor for anthology works, overflowing
primary_author's column — Work::insert() was failing silently (Catalog
insert methods never checked $wpdb->insert()'s return value), while the
following Edition/Isbn inserts for that same row went ahead anyway and
attached to whichever work_id was last successful. Fixed at the root:
inserts now throw on failure, the importer takes just the first author
(matching the column's actual semantic intent), and the import loop
catches per-row failures so one bad title can't abort a bulk run.
Verified end-to-end against a real 2000-title import: exact work/edition/
isbn/product count parity, no duplicates on re-run, real HTTP cart tests
for both shipping thresholds, live storefront search.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two real, verified findings from actually testing the "let Caddy issue a
self-signed cert automatically" approach against a literal IP:
1. TLS SNI is not sent for literal IP connections (out of spec — SNI's
HostName type explicitly excludes IPs). Confirmed via openssl s_client:
the handshake fails with a TLS-layer internal_error, even though Caddy
logs "certificate obtained successfully" — Caddy has no way to select a
cert without SNI. No Caddyfile config fixes this; the address has to be
a hostname.
2. Caddy's automatic-HTTPS heuristic only treats bare IPs and "localhost"
as obviously-private. Any other name — including a made-up LAN hostname
like "bookstore-staging.test" — it assumes might be real and tries
Let's Encrypt, which fails the same way staging.example.com did.
Fixed by forcing `tls internal` explicitly in Caddyfile.staging,
removing the guesswork entirely.
Verified end-to-end with curl --resolve (proper SNI, no real DNS/hosts
change needed to test): direct HTTPS 200, HTTP->HTTPS redirect chain 200.
README now documents the /etc/hosts requirement and how to trust Caddy's
root cert to skip the one-time browser warning.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two separate leaks were causing the same cosmetic-but-annoying warning to
survive every previous fix attempt:
1. Compose's own `secrets:` block reads the referenced file's *content*
as part of its config model, and applies the same interpolation
warning to it — even with DB_PASSWORD fully removed from every ${VAR}
and --env-file path. Switched from Compose's native `secrets:` to plain
bind mounts at the same /run/secrets/* paths: a bind mount only ever
touches the file's path, never its content, so it's immune. (Verified
this precisely with an isolated repro before rolling it out — the two
mechanisms behave differently even though they look equivalent.)
2. caddy's `env_file: .env.staging` (a leftover from the since-removed
basic-auth setup) loaded the *raw*, unfiltered env file directly,
bypassing deploy.sh/backup.sh/restore.sh's filtered-copy mechanism
entirely. Caddy only ever needed SITE_DOMAIN; switched to passing that
one value directly instead of the whole file.
Also fixed Caddyfile.staging: the site address had no explicit scheme, so
Caddy's automatic-HTTPS logic still applied to a private IP (registering
its own internal CA and redirecting HTTP->HTTPS) — not the "plain HTTP
only" behavior I'd assumed and told the user earlier. Prefixed with
`http://` to genuinely disable automatic HTTPS for this LAN-only box.
Verified end-to-end: fresh deploy with dollar-sign DB passwords produces
zero warnings, serves HTTP 200 on plain http:// with no redirect, and the
DB connection genuinely authenticates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bring-up wait only checked that php-cli runs, not that the official
image had finished copying WordPress's core files into the (possibly
freshly-emptied) wp_core volume — a step that takes a few real seconds on
first boot. On a fast pass through that window, wp-cli would run against
an empty /var/www/html and fail with "This does not seem to be a
WordPress installation," even though the container was otherwise healthy.
Now waits on wp-settings.php actually existing instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reported bug (a generated password containing "$BRjTx3tlmSpz" got
silently blanked, breaking the DB connection) is the same Compose
interpolation issue as the earlier bcrypt hash, but this time in fields
that are genuinely user-chosen and can't just be avoided by convention.
Switched DB_PASSWORD/DB_ROOT_PASSWORD to Docker's official `_FILE`
secrets convention (MARIADB_PASSWORD_FILE / WORDPRESS_DB_PASSWORD_FILE),
backed by Compose's native `secrets:` mechanism: deploy.sh writes the raw
value to secrets/<env>/db_password, and the container reads that file
directly — the value never passes through Compose's ${VAR} interpolation
at all. Verified end-to-end with an actual `$`-containing password,
including a full deploy → backup → restore → still-serving round trip.
Also fixed along the way (found while actually testing, not assumed):
- Makefile never exported ENVIRONMENT, so `make up` alone (bypassing
deploy.sh) would have left the new secrets path unresolved.
- deploy.sh chmod'd the secret files 600, unreadable by the container's
own UID (www-data) — fixed to 644, relying on the containing directory
(700) to keep other host users out instead.
- backup.sh/restore.sh still called `mysqldump`/`mysql`, which don't
exist in the mariadb:11 image under those names — renamed to
mariadb-dump/mariadb. (This means neither script had actually
succeeded before now; both are verified working end-to-end here.)
The supplier/payment API keys remain passed the old way — nothing reads
them yet (bookstore-core is still a stub), so there's no live bug to fix
there; noted in .env.example that the same _FILE pattern should be used
once that code exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Staging runs on a private-network VM, not reachable from outside, so the
HTTP basic-auth layer was unnecessary defense-in-depth. Removing it also
lets the Caddyfile drop the render-around-Compose workaround entirely
(that workaround existed specifically because Compose's interpolation
mangles a bcrypt hash) — the noindex header and blog_public=0 stay, since
those guard against search-engine indexing, a separate concern from
network-level access.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Replace the bare-repo post-receive hook with deploy/poll-deploy.sh: Gitea
and the Docker hosts are separate machines, so each box polls its branch
via host crontab instead of needing an exposed webhook receiver.
- Add Blocksy theme + Blocksy Companion auto-install to deploy.sh (free
tier; the paid Book Store starter site still needs a manual license step).
- Fix deploy.sh/backup.sh/restore.sh sourcing .env files as bash: a bcrypt
hash's `$2a$14$...` shape breaks under `set -u`. Replaced with
deploy/lib/env.sh, a literal (non-executing) KEY=VALUE reader.
- Fix docker compose itself mangling the same kind of value: both
`environment: ${VAR}` and `env_file:` run values through Compose's
interpolation, which silently blanks `$identifier`-shaped substrings.
The staging basic-auth hash is now rendered directly into the Caddyfile
by deploy.sh, bypassing Compose's variable system entirely.
- Fix dev/staging/production silently sharing one Compose project (and
therefore one db_data volume) by pinning an explicit -p per environment.
- cron and wordpress now share one environment anchor so they can't drift
apart again (cron was silently missing WORDPRESS_CONFIG_EXTRA before).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
twooey
