Imported deals become searchable: the automatic review of imports (#147)

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex be5d3ec..1483706 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -2,6 +2,8 @@
  
  ## Unreleased
  
++- Imported deals become searchable (the automatic review of imports): an offer imported from a configured feed source is checked when it is imported and passes when its link is https and points at that source's retailer domain, its title is not empty and at most 300 characters, its price is a positive number with a currency, it has not expired and it states none of the blocked/adult terms. A pass sets `review_state = auto_approved` (published, indexable, `reviewed_by = system:import-checks`); a fail stays `pending` with the failed check's name in `review_failed_check`. Offers people submit by hand keep today's review flow and a moderator can still hide any offer. `server/domain/import-review.js`, migration `0009_import_auto_review.sql`, `scripts/backfill-import-review.js` (batches of 500, `--dry-run`, counts only), `test/import-review.test.js`.
++
  - `llms.txt` lists only URLs that answer (SEO-34): the *Product JSON* link pointed at `/sitemaps/products.xml`, which no longer exists once the sitemap is an index, and `/sitemap.xml` itself was a single 24,908-URL file a crawler waited out. The link now names the index, and `test/pages.test.js` sweeps every URL in the file for an answer below 400.
  - The sitemaps are a bounded index (SEO-16): `GET /sitemap.xml` (openvibe-shared `seo.sitemapIndexXml`) names `/sitemap-pages.xml` and one `/sitemap-<n>.xml` child per 10,000 catalog URLs — every indexable offer, then every product, in the order the site has always listed them — so no child can pass a crawler's per-file limit and the union of the children is exactly the old catalog sitemap. A chunk number past the last answers 404; `robots.txt` still names `/sitemap.xml` and the content type and `cache-policy` headers are unchanged. `test/sitemap-build.test.js` seeds 20,005 offers and a product and pins all of it.
  - The front page is curated (`server/render/home.js`): a search and the site's numbers beside the hero, then editors' picks, hot, just in, ending soon, under $20, categories, shops and crowdfunding, how it works and six questions (FAQPage and ItemList JSON-LD). The hot list moved to `/hot` (`/?page=N` → 301); new `/ending`, `/picks`, `/c` and `/c/:category`; `/sitemaps/pages.xml` lists them all (the sitemap had no page a person browses). Imported deals keep their source's category and editors' pick (Sources feed@3; migration `0007_offer_category_pick.sql` adds `staff_pick`). Budgets: html 29.1 KB, css 24.2 KB.
ddiff --git a/README.md b/README.md
iindex 69bde2b..4bc3e1b 100644
---- a/README.md
++++ b/README.md
@@@ -253,6 +253,19 @@ when Sources fetched them again. A removed item (takedown, licence) removes its
  offer with no source left is disabled. Imported text is `review_state = pending` → `noindex` until a
  moderator records a review (a `usr_` subject; a service cannot review).
  
++**The automatic review of imports** (`server/domain/import-review.js`): an offer imported from a
++configured feed source is checked when it is imported. It passes when its link is https and points at
++that source's retailer domain (or a subdomain of it), its title is not empty and at most 300
++characters, its price is a positive number with a currency, it has not expired and its title and
++description state none of the blocked/adult terms. A pass sets `review_state = auto_approved`
++(published, indexable, `reviewed_by = system:import-checks`); a fail stays `pending` with the failed
++check's name in `review_failed_check`. The configured sources and their retailer domains are in
++`server/config.js` (`DEALS_IMPORT_REVIEW_SOURCES` replaces them, `DEALS_IMPORT_REVIEW=off` turns the
++review off); a source that is not configured is never auto-approved. The offers imported before this
++existed are checked by `scripts/backfill-import-review.js` (`--dry-run` prints counts only).
++Offers people submit by hand are not touched: their review flow is unchanged, and a moderator can
++still hide any offer.
++
  The crowdfunding feed (`category=crowdfunding`, server/domain/campaigns.js) is pulled through the same chassis with its
  own cursor (`sources.crowdfunding.after` in `deals_ingest_cursor`) into `deal_campaigns` (migration
  `0005_crowdfunding.sql`), never into offers: name, summary, type, first tag, currency, the total received and yearly
ddiff --git a/migrations/0009_import_auto_review.sql b/migrations/0009_import_auto_review.sql
nnew file mode 100644
iindex 0000000..d6d51c8
---- /dev/null
++++ b/migrations/0009_import_auto_review.sql
@@@ -0,0 +1,12 @@
++-- phase: expand
++-- The automatic review of imported offers (server/domain/import-review.js): an offer imported from a
++-- configured feed source is checked when it is imported, and the existing imports are checked by
++-- scripts/backfill-import-review.js. A pass sets review_state = 'auto_approved' (published and
++-- indexable, reviewed_by = 'system:import-checks'); a fail stays 'pending' and the failed check's
++-- name is kept in review_failed_check. Offers people submit by hand are not touched: their review
++-- flow is unchanged.
++ALTER TABLE deal_offers DROP CONSTRAINT IF EXISTS deal_offers_review_state_check;
++ALTER TABLE deal_offers ADD CONSTRAINT deal_offers_review_state_check
++    CHECK (review_state IN ('not_required','pending','reviewed','auto_approved'));
++ALTER TABLE deal_offers ADD COLUMN IF NOT EXISTS review_failed_check text;
++CREATE INDEX IF NOT EXISTS deal_offers_pending_import ON deal_offers (id) WHERE review_state = 'pending' AND origin = 'import';
ddiff --git a/scripts/backfill-import-review.js b/scripts/backfill-import-review.js
nnew file mode 100644
iindex 0000000..44f5a40
---- /dev/null
++++ b/scripts/backfill-import-review.js
@@@ -0,0 +1,123 @@
++#!/usr/bin/env node
++'use strict';
++
++/**
++ * The automatic review of the offers imported before it existed (server/domain/import-review.js).
++ *
++ * The same checks the importer runs now, over the offers still waiting for a person's review:
++ * an offer from a configured feed source passes when its link is https and points at that source's
++ * retailer domain, its title is present and short, its price is a positive number with a currency,
++ * it has not expired and it states no blocked term. A pass sets review_state = 'auto_approved'
++ * (published, indexable, reviewed by 'system:import-checks'); a fail stays pending with the failed
++ * check's name on the offer.
++ *
++ *   node scripts/backfill-import-review.js --dry-run      # counts only, nothing is written
++ *   node scripts/backfill-import-review.js                 # in batches of 500
++ *   node scripts/backfill-import-review.js --batch-size 500 --limit 1000 --no-reindex
++ *
++ * It prints counts only (passed, failed per check) — never an offer's content. It is idempotent:
++ * an offer that passed is not pending any more, and an offer that failed is checked again and
++ * fails the same way.
++ */
++
++const configLib = require('../server/config');
++const { openStore } = require('../server/db');
++const { createServiceOutbox } = require('openvibe-sdk/events');
++const { createReads } = require('../server/domain/reads');
++const { createCatalog } = require('../server/domain/catalog');
++const { createPublication } = require('../server/domain/publication');
++const { createIndexing } = require('../server/domain/indexing');
++const { evaluateImport, retailerDomainsFor, CHECKS } = require('../server/domain/import-review');
++
++const AUTO_REVIEWER = 'system:import-checks';
++
++function arg(name, def) {
++    const hit = process.argv.find((a) => a === `--${name}` || a.startsWith(`--${name}=`));
++    if (!hit) return def;
++    if (hit.includes('=')) return hit.split('=').slice(1).join('=');
++    const next = process.argv[process.argv.indexOf(hit) + 1];
++    return next && !next.startsWith('--') ? next : true;
++}
++
++const BATCH = Math.max(1, parseInt(arg('batch-size', 500), 10) || 500);
++const LIMIT = arg('limit', null) ? Math.max(0, parseInt(arg('limit', 0), 10) || 0) : null;
++const DRY = Boolean(arg('dry-run', false));
++const REINDEX = !arg('no-reindex', false);
++
++const BATCH_SQL = `SELECT f.id, f.title, f.description, f.url, f.expires_at, f.status, s.source_key, o.price_num, o.currency
++  FROM deal_offers f
++  JOIN LATERAL (SELECT source_key FROM deal_offer_sources WHERE offer_id = f.id AND kind = 'sources_item' AND removed_at IS NULL ORDER BY id LIMIT 1) s ON TRUE
++  LEFT JOIN LATERAL (SELECT price_num, currency FROM deal_price_observations WHERE offer_id = f.id ORDER BY observed_at DESC, seq DESC LIMIT 1) o ON TRUE
++ WHERE f.origin = 'import' AND f.review_state = 'pending' AND f.id > ?
++ ORDER BY f.id
++ LIMIT ?`;
++
++(async () => {
++    const config = configLib.load();
++    if (!config.importReview || !config.importReview.enabled) {
++        console.log(JSON.stringify({ error: 'the automatic review is off (DEALS_IMPORT_REVIEW=off)' }));
++        process.exit(1);
++    }
++    const log = { log() {}, warn: (...a) => console.error(...a), error: (...a) => console.error(...a) };
++    const store = await openStore(config, { log });
++    const outbox = createServiceOutbox({
++        db: store.db, source: 'deals', eventsUrl: config.events.url, networkInternalUrl: config.networkInternalUrl,
++        clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
++        intervalMs: config.events.intervalMs, now: store.now, log,
++    });
++    const reads = createReads({ store });
++    const catalog = createCatalog({ store });
++    const publication = createPublication({ config, store, reads, catalog });
++    const indexing = createIndexing({ config, store, publication, outbox, catalog, indexnow: null });
++
++    const counts = { scanned: 0, passed: 0, failed: 0, skipped_source: 0, reindexed: 0 };
++    counts.failed_by_check = Object.fromEntries(CHECKS.map((c) => [c, 0]));
++    let cursor = '';
++    const now = () => store.now();
++
++    try {
++        for (;;) {
++            const rows = await store.db.prepare(BATCH_SQL).all(cursor, BATCH);
++            if (!rows.length) break;
++            for (const row of rows) {
++                cursor = row.id;
++                counts.scanned++;
++                const domains = retailerDomainsFor(config.importReview, row.source_key);
++                if (!domains) { counts.skipped_source++; continue; }
++                const result = evaluateImport({
++                    url: row.url, title: row.title, description: row.description,
++                    price_num: row.price_num, currency: row.currency,
++                    expires_at: row.expires_at, status: row.status,
++                }, { retailerDomains: domains, blockedTerms: config.importReview.blockedTerms, now: now() });
++                if (result.ok) {
++                    counts.passed++;
++                    if (!DRY) {
++                        await store.tx(async () => {
++                            const at = now();
++                            await store.db.prepare("UPDATE deal_offers SET review_state = 'auto_approved', reviewed_by = ?, reviewed_at = ?, review_failed_check = NULL, updated_at = ? WHERE id = ? AND review_state = 'pending'")
++                                .run(AUTO_REVIEWER, at, at, row.id);
++                            await store.db.prepare('INSERT INTO moderation_log (action, offer_id, target_id, actor, reason, before, after, at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)')
++                                .run('auto_review', row.id, null, AUTO_REVIEWER, 'the import checks passed', JSON.stringify({ review_state: 'pending' }), JSON.stringify({ review_state: 'auto_approved' }), at);
++                            if (REINDEX) { await indexing.indexOffer(await reads.get(row.id)); counts.reindexed++; }
++                        });
++                    }
++                } else {
++                    counts.failed++;
++                    counts.failed_by_check[result.failed] = (counts.failed_by_check[result.failed] || 0) + 1;
++                    if (!DRY) {
++                        await store.tx(async () => {
++                            await store.db.prepare("UPDATE deal_offers SET review_failed_check = ?, updated_at = ? WHERE id = ? AND review_state = 'pending'").run(result.failed, now(), row.id);
++                        });
++                    }
++                }
++                if (LIMIT && counts.scanned >= LIMIT) { cursor = '￿'; break; }
++            }
++            if (LIMIT && counts.scanned >= LIMIT) break;
++        }
++    } finally {
++        if (outbox.stop) await outbox.stop();
++        await store.close();
++    }
++    console.log(JSON.stringify({ dry_run: DRY, batch_size: BATCH, ...counts }));
++    process.exit(0);
++})().catch((err) => { console.error(err && err.message ? err.message : err); process.exit(1); });
ddiff --git a/server/config.js b/server/config.js
iindex cf04370..924ab96 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -13,8 +13,29 @@ const trim = (s) => String(s || '').replace(/\/+$/, '');
  const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
  const num = (v, def) => (v != null && v !== '' && Number.isFinite(Number(v)) ? Number(v) : def);
  const list = (v) => String(v || '').split(',').map((s) => s.trim()).filter(Boolean);
++const json = (v, def) => { try { const p = v ? JSON.parse(v) : null; return p && typeof p === 'object' ? p : def; } catch { return def; } };
  const HOUR = 60 * 60 * 1000;
  
++/**
++ * The feed sources whose imported offers may be reviewed automatically, and the retailer domain
++ * each of them links to. A source that is not listed here is never auto-approved.
++ */
++const IMPORT_REVIEW_SOURCES = Object.freeze({
++    'dealnews-daily': { retailerDomains: ['dealnews.com'] },
++    'crawl-zappos-com': { retailerDomains: ['zappos.com'] },
++    'crawl-footlocker-com': { retailerDomains: ['footlocker.com'] },
++    'crawl-anker-com': { retailerDomains: ['anker.com'] },
++    'crawl-humblebundle-com': { retailerDomains: ['humblebundle.com'] },
++    'crawl-gog-com': { retailerDomains: ['gog.com'] },
++    'crawl-allbirds-com': { retailerDomains: ['allbirds.com'] },
++});
++
++/** The blocked/adult terms an imported offer's title and description may not contain. */
++const IMPORT_REVIEW_BLOCKED_TERMS = Object.freeze([
++    'porn', 'xxx video', 'xxxvideo', 'hentai', 'nsfw', 'onlyfans', 'camgirl', 'escort service',
++    'viagra', 'cialis', 'payday loan', 'free bitcoin', 'crypto giveaway', 'phishing', 'work from home scam',
++]);
++
  function load(env = process.env) {
      const nodeEnv = env.NODE_ENV || 'development';
      const isProduction = nodeEnv === 'production';
@@@ -101,6 +122,17 @@ function load(env = process.env) {
              refreshBatch: int(env.DEALS_IMPORT_REFRESH_BATCH, 50),
          },
  
++        // The automatic review of imported offers (server/domain/import-review.js). Only a source
++        // listed here is ever auto-approved, and only the retailer domains listed for it. A crawl
++        // source's key names its retailer (crawl-zappos-com -> zappos.com).
++        importReview: {
++            enabled: env.DEALS_IMPORT_REVIEW !== 'off',
++            sources: json(env.DEALS_IMPORT_REVIEW_SOURCES, IMPORT_REVIEW_SOURCES),
++            // A short list (there was none before): adult terms and the spam terms an import may
++            // never carry. DEALS_IMPORT_REVIEW_BLOCKED_TERMS replaces it.
++            blockedTerms: env.DEALS_IMPORT_REVIEW_BLOCKED_TERMS ? list(env.DEALS_IMPORT_REVIEW_BLOCKED_TERMS) : IMPORT_REVIEW_BLOCKED_TERMS.slice(),
++        },
++
          // OpenVibe.Events: the outbox relay runs only when EVENTS_URL and the client secret are set;
          // the webhook consumer (/internal/events) only when DEALS_EVENTS_SECRET is set.
          events: {
ddiff --git a/server/domain/import-review.js b/server/domain/import-review.js
nnew file mode 100644
iindex 0000000..3b32a9a
---- /dev/null
++++ b/server/domain/import-review.js
@@@ -0,0 +1,83 @@
++'use strict';
++
++/**
++ * The automatic review of imported offers.
++ *
++ * An offer imported from a configured feed source (a Sources `source_key` in the config) is checked
++ * when it is imported, and the offers imported before this existed are checked by
++ * scripts/backfill-import-review.js. A pass sets review_state = 'auto_approved': the offer is
++ * published and indexable and records reviewed_by = 'system:import-checks'. A fail stays 'pending'
++ * and the failed check's name is kept on the offer (review_failed_check), so a moderator sees why.
++ *
++ * The checks (the first that fails is the one that is recorded):
++ *
++ *   url_scheme     the link is https
++ *   url_domain     the link's host is the source's retailer domain, or a subdomain of it
++ *   title          the title is not empty and at most 300 characters
++ *   price          the price is a positive number and states a currency
++ *   expiry         the offer has not expired
++ *   blocked_terms  the title and the description state none of the blocked/adult terms
++ *
++ * Offers people submit by hand never come here: their review flow is unchanged.
++ */
++
++const CHECKS = Object.freeze(['url_scheme', 'url_domain', 'title', 'price', 'expiry', 'blocked_terms']);
++
++/** The retailer domains of a source, or null when the source is not configured (never auto-approved). */
++function retailerDomainsFor(config, sourceKey) {
++    const section = (config && config.importReview) || config || {};
++    const sources = section.sources || {};
++    const entry = sourceKey && Object.prototype.hasOwnProperty.call(sources, sourceKey) ? sources[sourceKey] : null;
++    if (!entry) return null;
++    const domains = (Array.isArray(entry) ? entry : entry.retailerDomains || []).map((d) => String(d || '').trim().toLowerCase()).filter(Boolean);
++    return domains.length ? domains : null;
++}
++
++/** True when the host is the retailer domain itself or a subdomain of it. */
++function hostAllowed(host, domains) {
++    const h = String(host || '').toLowerCase();
++    if (!h) return false;
++    return (domains || []).some((d) => h === d || h.endsWith(`.${d}`));
++}
++
++/** The lower-cased title and description a blocked term is looked for in. */
++function blockedIn(text, terms) {
++    const hay = String(text || '').toLowerCase();
++    if (!hay) return null;
++    for (const term of terms || []) {
++        const t = String(term || '').trim().toLowerCase();
++        if (t && hay.includes(t)) return t;
++    }
++    return null;
++}
++
++/**
++ * The checks, one by one. Returns { ok, failed } — `failed` is the name of the first check that
++ * did not pass (null when everything passed).
++ */
++function evaluateImport(offer, { retailerDomains = [], blockedTerms = [], now = Date.now() } = {}) {
++    const fail = (name) => ({ ok: false, failed: name });
++
++    let url = null;
++    try { url = new URL(String(offer.url || '')); } catch { return fail('url_scheme'); }
++    if (url.protocol !== 'https:') return fail('url_scheme');
++    if (!hostAllowed(url.hostname, retailerDomains)) return fail('url_domain');
++
++    const title = String(offer.title == null ? '' : offer.title).trim();
++    if (!title || title.length > 300) return fail('title');
++
++    const price = Number(offer.price_num);
++    const currency = typeof offer.currency === 'string' ? offer.currency.trim() : '';
++    if (!Number.isFinite(price) || price <= 0 || !currency) return fail('price');
++
++    const expiresAt = offer.expires_at == null ? null : Number(offer.expires_at);
++    const expired = offer.status === 'expired' || (Number.isFinite(expiresAt) && expiresAt <= now);
++    if (expired) return fail('expiry');
++
++    const term = blockedIn(`${offer.title || ''} ${offer.description || ''}`, blockedTerms);
++    if (term) return fail('blocked_terms');
++
++    return { ok: true, failed: null };
++}
++
++module.exports = { CHECKS, evaluateImport, retailerDomainsFor, hostAllowed, blockedIn };
ddiff --git a/server/domain/offers.js b/server/domain/offers.js
iindex 26230a6..1753f8f 100644
---- a/server/domain/offers.js
++++ b/server/domain/offers.js
@@@ -19,6 +19,10 @@ const {
      text, longText, CONDITIONS, AVAILABILITY,
  } = require('./values');
  const { isDeal } = require('./price-trend');
++const { evaluateImport, retailerDomainsFor } = require('./import-review');
++
++/** The reviewer of an imported offer that passed the automatic checks (server/domain/import-review.js). */
++const AUTO_REVIEWER = 'system:import-checks';
  
  const FUTURE_SKEW_MS = 5 * 60 * 1000;
  
@@@ -187,6 +191,34 @@ function createOffers(deps) {
          });
      }
  
++    /**
++     * The automatic review of an imported offer (server/domain/import-review.js). Only an offer from
++     * a configured feed source is reviewed; it passes when its link is https and points at that
++     * source's retailer domain, its title is present and short, its price is a positive number with
++     * a currency, it has not expired and it states no blocked term. A pass publishes it (reviewed by
++     * 'system:import-checks'); a fail stays pending with the failed check's name on the offer.
++     */
++    async function autoReview(offer, source, obsFields) {
++        const cfg = config.importReview;
++        if (!cfg || !cfg.enabled) return null;
++        const domains = retailerDomainsFor(cfg, source && source.source_key);
++        if (!domains) return null;
++        const now = store.now();
++        const result = evaluateImport({
++            url: offer.url, title: offer.title, description: offer.description,
++            price_num: obsFields ? obsFields.price_num : null, currency: obsFields ? obsFields.currency : null,
++            expires_at: offer.expires_at, status: offer.status,
++        }, { retailerDomains: domains, blockedTerms: cfg.blockedTerms, now });
++        if (result.ok) {
++            await db.prepare("UPDATE deal_offers SET review_state = 'auto_approved', reviewed_by = ?, reviewed_at = ?, review_failed_check = NULL, updated_at = ? WHERE id = ?")
++                .run(AUTO_REVIEWER, now, now, offer.id);
++            await logAction('auto_review', { offerId: offer.id, actor: AUTO_REVIEWER, reason: 'the import checks passed', before: { review_state: 'pending' }, after: { review_state: 'auto_approved' } });
++        } else {
++            await db.prepare("UPDATE deal_offers SET review_failed_check = ?, updated_at = ? WHERE id = ? AND review_state = 'pending'").run(result.failed, now, offer.id);
++        }
++        return result;
++    }
++
      /** Created by the Sources importer (no person); text is third-party and waits for review. */
      async function createImported(f, source, obsFields, { observedAt, sourceRevision }) {
          const st = await catalog.ensureStore(domainOf(f.url), f.store_name || null);
@@@ -198,6 +230,7 @@ function createOffers(deps) {
          const src = await ensureSource(offer.id, source);
          await hotness.snapshot(offer.id, 'create');
          await recordObservation(offer.id, src.id, obsFields, { origin: 'import', observedAt, sourceRevision });
++        await autoReview(offer, source, obsFields);
          await indexing.emitOffer('deals.offer.created', offer, { actor: 'svc:deals', extra: { imported_from: { service: 'sources', type: 'item', id: source.ref_id } } });
          await indexing.reindex(await reads.get(offer.id));
          return await reads.get(offer.id);
ddiff --git a/test/import-review.test.js b/test/import-review.test.js
nnew file mode 100644
iindex 0000000..e1eb4fd
---- /dev/null
++++ b/test/import-review.test.js
@@@ -0,0 +1,107 @@
++'use strict';
++/**
++ * The automatic review of imported offers (server/domain/import-review.js): an offer imported from a
++ * configured feed source is checked when it is imported, and a hand-submitted offer keeps today's
++ * review flow.
++ */
++const assert = require('assert');
++const { boot, check, done, robotsOf } = require('./helpers/boot');
++const { sourceItem } = require('./helpers/mocks');
++const { evaluateImport } = require('../server/domain/import-review');
++
++const RETAILER = 'https://www.zappos.com';
++
++(async () => {
++    const t = await boot();
++    const alice = t.network.addUser('alice');
++    const byUrl = async (u) => await t.ctx.store.db.prepare('SELECT * FROM deal_offers WHERE url = ?').get(u);
++    let n = 0;
++    const item = (over) => sourceItem({
++        id: `itm_01K5AAAAAAAAAAAAAAAAAAAAA${String(++n).padStart(2, '0')}`,
++        sourceKey: 'crawl-zappos-com', title: 'MSI Vigor GK30 gaming keyboard', url: `${RETAILER}/p/${n}`,
++        retrievedAt: t.clock.now(), fields: { price: '35.00', currency: 'USD' }, ...over,
++    });
++
++    await check('an import from a configured source passes the checks: it is published, indexable and reviewed by the system', async () => {
++        t.sources.put(item({}));
++        await t.ctx.importer.pull();
++        const offer = await byUrl(`${RETAILER}/p/1`);
++        assert.strictEqual(offer.review_state, 'auto_approved');
++        assert.strictEqual(offer.reviewed_by, 'system:import-checks');
++        assert.ok(offer.reviewed_at >= t.clock.now(), 'the time of the review is recorded');
++        assert.strictEqual(offer.review_failed_check, null);
++        const json = await t.offerJson(offer.slug);
++        assert.strictEqual(json.indexability.indexable, true);
++        assert.notStrictEqual(robotsOf((await t.get(`/d/${offer.slug}`)).text), 'noindex, follow');
++        const doc = (await t.events('deals.index_document.upserted')).filter((e) => e.payload.id === offer.id).pop().payload;
++        assert.strictEqual(doc.indexability.decision, 'index');
++    });
++
++    await check('the url_scheme check fails: a link that is not https stays pending', async () => {
++        t.sources.put(item({ url: 'http://www.zappos.com/p/http' }));
++        await t.ctx.importer.pull();
++        const offer = await byUrl('http://www.zappos.com/p/http');
++        assert.strictEqual(offer.review_state, 'pending');
++        assert.strictEqual(offer.review_failed_check, 'url_scheme');
++        assert.strictEqual(offer.reviewed_by, null);
++    });
++
++    await check('the url_domain check fails: a link outside the retailer domain stays pending', async () => {
++        t.sources.put(item({ url: 'https://other-shop.example/p/other' }));
++        await t.ctx.importer.pull();
++        const offer = await byUrl('https://other-shop.example/p/other');
++        assert.strictEqual(offer.review_state, 'pending');
++        assert.strictEqual(offer.review_failed_check, 'url_domain');
++    });
++
++    await check('the title check fails: an empty title or one over 300 characters is not published', async () => {
++        const base = { url: 'https://www.zappos.com/p/title', price_num: 35, currency: 'USD' };
++        assert.strictEqual(evaluateImport({ ...base, title: '   ' }, { retailerDomains: ['zappos.com'] }).failed, 'title');
++        assert.strictEqual(evaluateImport({ ...base, title: 'x'.repeat(301) }, { retailerDomains: ['zappos.com'] }).failed, 'title');
++        assert.strictEqual(evaluateImport({ ...base, title: 'x'.repeat(300) }, { retailerDomains: ['zappos.com'] }).failed, null);
++    });
++
++    await check('the price check fails: a price that is missing, not positive or without a currency stays pending', async () => {
++        t.sources.put(item({ url: `${RETAILER}/p/noprice`, fields: { price: null, currency: null } }));
++        await t.ctx.importer.pull();
++        const offer = await byUrl(`${RETAILER}/p/noprice`);
++        assert.strictEqual(offer.review_state, 'pending');
++        assert.strictEqual(offer.review_failed_check, 'price');
++        assert.strictEqual(evaluateImport({ url: 'https://www.zappos.com/p/x', title: 'A thing', price_num: 0, currency: 'USD' }, { retailerDomains: ['zappos.com'] }).failed, 'price');
++        assert.strictEqual(evaluateImport({ url: 'https://www.zappos.com/p/x', title: 'A thing', price_num: 10, currency: null }, { retailerDomains: ['zappos.com'] }).failed, 'price');
++    });
++
++    await check('the expiry check fails: an offer that has expired is not published', async () => {
++        const base = { url: 'https://www.zappos.com/p/exp', title: 'A thing', price_num: 10, currency: 'USD' };
++        assert.strictEqual(evaluateImport({ ...base, status: 'expired' }, { retailerDomains: ['zappos.com'] }).failed, 'expiry');
++        assert.strictEqual(evaluateImport({ ...base, expires_at: 1000 }, { retailerDomains: ['zappos.com'], now: 2000 }).failed, 'expiry');
++        assert.strictEqual(evaluateImport({ ...base, expires_at: 3000 }, { retailerDomains: ['zappos.com'], now: 2000 }).failed, null);
++    });
++
++    await check('the blocked_terms check fails: a title that states a blocked term stays pending', async () => {
++        t.sources.put(item({ url: `${RETAILER}/p/blocked`, title: 'Free porn video download' }));
++        await t.ctx.importer.pull();
++        const offer = await byUrl(`${RETAILER}/p/blocked`);
++        assert.strictEqual(offer.review_state, 'pending');
++        assert.strictEqual(offer.review_failed_check, 'blocked_terms');
++    });
++
++    await check('a source that is not configured is never auto-approved', async () => {
++        t.sources.put(item({ url: `${RETAILER}/p/unconfigured`, sourceKey: 'shop-jsonld' }));
++        await t.ctx.importer.pull();
++        const offer = await byUrl(`${RETAILER}/p/unconfigured`);
++        assert.strictEqual(offer.review_state, 'pending');
++        assert.strictEqual(offer.review_failed_check, null);
++    });
++
++    await check('a hand-submitted offer keeps today\'s review flow: no automatic review', async () => {
++        const offer = await t.submit(alice, { title: 'A keyboard I found', url: 'https://shop.example/p/mine', price: '20.00', currency: 'USD' });
++        assert.strictEqual(offer.review_state, 'not_required');
++        assert.strictEqual(offer.reviewed_by, null);
++        const json = await t.offerJson(offer.slug);
++        assert.strictEqual(json.indexability.indexable, true);
++    });
++
++    await t.close();
++    done();
++})().catch((e) => { console.error(e); process.exit(1); });