openvibers/openvibe.deals Commits
Deals: per-source policy — DealNews links kept verbatim, attributed on card/page/feeds, never served to a browser extension (its feed terms). (#11)
Changes
@@@ -30,7 +30,7 @@ The nine charter tables live in Deals' own PostgreSQL database (`ov_deals` on th|---|---|| `deal_products` | product records (name, brand, category), `/p/:slug` || `deal_product_aliases` | `gtin` (normalised to 14 digits), `mpn`, `sku`, exact `name`, `url` → one product; an alias never silently moves to another product |--| `deal_offers` | an offer: title, link (tracking parameters stripped), store, product, status (`active`/`expired`/`disabled`), **stated** expiry, review state, and the merge pointer `merged_into` |++| `deal_offers` | an offer: title, link (tracking parameters stripped, except where a source policy keeps it verbatim — see "Per-source policy"), store, product, status (`active`/`expired`/`disabled`), **stated** expiry, review state, and the merge pointer `merged_into` || `deal_offer_sources` | where an offer came from: the submission, a person's observations, or an OpenVibe.Sources item (typed reference `{sources, item, itm_…, revision}`; the item is never copied) || `deal_votes` | one row per (offer, subject): `+1` / `-1` / `0` (removed, kept as history), with the weight and IP hash it was cast with || `deal_hotness_snapshots` | every hotness computation with all of its inputs (formula `hot@1`) |@@@ -135,7 +135,8 @@ Consumed: `sources.item.created|updated|removed` at `POST /internal/events` (sig- The same link (normalised: fragment and tracking/affiliate parameters removed, `www.` and schemeignored) cannot be posted twice: a person gets the existing deal (409 / redirect), a second Sources-- item attaches to it as another source.++ item attaches to it as another source. A link a source policy keeps verbatim is identified as++ received, so it never strips parameters to collide with a normalised one.- Moderators merge listings of one deal (`/mod`, "possible duplicates" = same store and same productor title). **Merging moves nothing:** B gets `merged_into = A`; A's page, JSON, votes, history andsources read across the group; B's URL answers 301. A person who voted on both counts **once**@@@ -227,6 +228,37 @@ when Sources fetched them again. A removed item (takedown, licence) removes itsoffer with no source left is disabled. Imported text is `review_state = pending` → `noindex` until amoderator records a review (a `usr_` subject; a service cannot review).++### Per-source policy: verbatim links, attribution and browser extensions++++Deals normalises an imported offer's link by default: the fragment and tracking/affiliate parameters++are removed so two posts of one deal share an identity. A source whose terms require otherwise gets++an explicit entry in `server/domain/source-policy.js`, keyed by the Sources `source_key` — the prose++`license_note` / `terms_note` are never parsed, the map is data:++++| Policy field | Effect |++|---|---|++| `verbatimLinks` | the link is stored and shown exactly as received (no fragment or tracking-parameter stripping, no rewriting, no `rel` change that alters the URL), and the title is used as the source stated it, with no added words |++| `attribution` | `"Deal via <text>"` linking to `url` is shown on the offer card, the offer page and the product comparison, and carried in the offer's JSON (`.json` / API `attribution`) and its RSS/Atom/JSON-Feed summary and tags |++| `noExtension` | the offer is excluded from the API when the caller's `Origin` is a browser-extension scheme (`chrome-extension://`, `moz-extension://`, …): absent from `GET /offers`, `GET /products/:slug` and `GET /stores/:domain`, and `GET /offers/:id` answers 404 |++++**DealNews** ([feed terms](https://www.dealnews.com/pages/rss.html), checked 2026-10-07) allows public++use of its feeds only without removing or adding content within the feed display, without modifying++the links or referral codes, with attribution to DealNews as text (e.g. "DealNews") or their 88x31++logo, and never inside a browser extension. `dealnews-daily` therefore sets all three:++++```js++'dealnews-daily': {++ verbatimLinks: true,++ attribution: { text: 'DealNews', url: 'https://www.dealnews.com/' },++ noExtension: true,++},++```++++Community submissions and every source with no entry keep today's behaviour (normalised links, no++attribution, visible to every caller). `test/import.test.js` and `test/api.test.js` cover the verbatim++link byte-for-byte, the attribution on card, page and JSON, the untouched default, and the++extension-facing exclusion.++## Discoverability (roadmap §32)- SSR HTML with canonical, Open Graph, JSON-LD (`Product`/`Offer`, `BreadcrumbList`, `WebSite` with@@@ -262,7 +294,8 @@ render time, so the cache stays short); signed-in views, forms, API responses, e- **OpenVibe.Network:** SSO (OAuth client `deals`, registered in production), JWKS,`identity.subject.resolve`.- **OpenVibe.Sources:** `sources.item.read`; a `deals`-category source must be registered and enabled-- there (the seeded `dealnews-daily` is disabled until a person re-verifies its terms).++ there (the seeded `dealnews-daily` is disabled until a person re-verifies its terms; its feed terms++ are now encoded in `server/domain/source-policy.js`, so enabling it stays within them).- **OpenVibe.Community:** `community.comment.write`, optionally `community.comment.moderate`.- **OpenVibe.Events:** `events.event.publish`; `events.subscription.manage` once, for`scripts/subscribe.js`.@@@ -20,6 +20,7 @@*/const seo = require('openvibe-publishing/seo');const { iso } = require('./values');++const { attributionOf, hidesFromExtension } = require('./source-policy');function createPublication({ config, store, reads, catalog }) {const abs = (p) => seo.canonicalUrl(config.baseUrl, p);@@@ -65,6 +66,7 @@ function createPublication({ config, store, reads, catalog }) {const latest = await reads.latestObservation(ids);const tally = await reads.tally(ids);const snap = await reads.lastSnapshot(root.id);++ const sources = await reads.sources(ids);return {root,ids,@@@ -73,7 +75,11 @@ function createPublication({ config, store, reads, catalog }) {observations: await reads.observations(ids),latest,freshness: freshness(latest, now),-- sources: await reads.sources(ids),++ sources,++ // A source policy may require attribution (shown on cards, the page, JSON and feeds) and++ // may forbid the offer's content in a browser extension (DealNews' feed terms).++ attribution: attributionOf(sources),++ extensionExcluded: hidesFromExtension(sources),members: await Promise.all(ids.filter((id) => id !== root.id).map(async (id) => await reads.get(id))),tally,hotness: snap,@@@ -107,6 +113,7 @@ function createPublication({ config, store, reads, catalog }) {// "latest", never "current": whether it still holds is what freshness says.latest_observation: v.latest ? observationDto(v.latest) : null,freshness: v.freshness,++ attribution: v.attribution || null,observations: v.observations.map(observationDto),sources: v.sources.map((s) => ({id: s.id, kind: s.kind, listing: s.offer_id,@@@ -22,6 +22,10 @@* headline — it stays "not stated" until a structured source or a person states it* other kinds skipped*++ * A source policy (server/domain/source-policy.js) may require a source's links kept verbatim and++ * name an attribution (DealNews' feed terms); such a source's links are stored and shown exactly as++ * received, its title is used as stated, and the attribution is shown with every offer it touches.++ ** observed_at = the item's provenance.retrieved_at (when Sources saw it). An item Deals already* imported produces a new observation only when its revision or its retrieved_at advanced, so* replays are no-ops. Two items (or a submission and an item) with the same link attach to the same@@@ -33,8 +37,9 @@*/const { createChangeCursor, pullChanges } = require('openvibe-publishing/ingest');const {-- normalizeUrl, urlKey, parseAmount, SCHEMA_AVAILABILITY, SCHEMA_CONDITION, iso,++ normalizeUrl, verbatimUrl, urlKey, parseAmount, SCHEMA_AVAILABILITY, SCHEMA_CONDITION, iso,} = require('./values');++const { keepsLinksVerbatim } = require('./source-policy');const CURSOR = 'sources.deals.after';@@@ -88,8 +93,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourcfunction candidates(item) {const f = item.fields || {};const stated = (t) => { const ms = t ? Date.parse(t) : NaN; return Number.isFinite(ms) ? ms : null; };++ // A source policy may require its links to be kept exactly as received (DealNews' feed++ // terms); every other source keeps today's normalisation.++ const verbatim = keepsLinksVerbatim(item.source_key);++ const linkOf = (raw) => (verbatim ? verbatimUrl(raw) : normalizeUrl(raw));if (item.kind === 'offer') {-- const url = normalizeUrl(f.url || item.canonical_url);++ const url = linkOf(f.url || item.canonical_url);if (!url || !item.title) return [];return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields(f), expires_at: stated(f.valid_until), store_name: f.seller || null }];}@@@ -98,11 +107,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourcconst seen = new Set();const out = [];for (const o of (Array.isArray(f.offers) ? f.offers : []).slice(0, 20)) {-- const url = normalizeUrl(o.url || item.canonical_url);++ const url = linkOf(o.url || item.canonical_url);if (!url || seen.has(urlKey(url))) continue;seen.add(urlKey(url));out.push({-- part: urlKey(url), url, title: o.seller ? `${item.title} — ${o.seller}` : item.title, description: item.summary,++ // A verbatim source's title is used as stated, with no added words.++ part: urlKey(url), url, title: !verbatim && o.seller ? `${item.title} — ${o.seller}` : item.title, description: item.summary,obs: obsFields(o), expires_at: stated(o.valid_until), store_name: o.seller || null,product: { name: item.title, brand: f.brand, gtin: f.gtin, mpn: f.mpn, sku: f.sku },});@@@ -110,7 +120,7 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourcreturn out;}if (item.kind === 'article' || item.kind === 'record') {-- const url = normalizeUrl(item.canonical_url);++ const url = linkOf(item.canonical_url);if (!url || !item.title) return [];return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields({}), expires_at: null, store_name: null }];}nnew file mode 100644@@@ -0,0 +1,57 @@++'use strict';++++/**++ * Per-source policy for imported offers: which links must be kept exactly as the source stated them,++ * how the source is attributed, and whether its content may reach a browser extension.++ *++ * A source that hands Deals a feed with terms gets an entry here, explicitly, keyed by the++ * OpenVibe.Sources `source_key` (the key Sources labels each item with, e.g. `dealnews-daily`).++ * Sources' `license_note` / `terms_note` are prose and are never parsed: a source with no entry gets++ * today's default — links normalised (tracking and affiliate parameters stripped) and no attribution.++ *++ * DealNews' feed terms (https://www.dealnews.com/pages/rss.html) allow public use of their feeds only++ * (1) without removing or adding content within the feed display, (2) without modifying the links or++ * referral codes, (3) with attribution to DealNews as text (e.g. "DealNews") or with their 88x31 logo,++ * and (4) never inside a browser extension. `dealnews-daily` therefore keeps its links verbatim, is++ * attributed wherever the offer appears, and is kept out of the API a browser extension could read.++ */++++const POLICIES = Object.freeze({++ 'dealnews-daily': Object.freeze({++ verbatimLinks: true,++ attribution: Object.freeze({ text: 'DealNews', url: 'https://www.dealnews.com/' }),++ // Terms (4): the feed may never be used inside a browser extension.++ noExtension: true,++ }),++});++++/** The policy for a Sources `source_key`, or null (today's default: normalise the link, no attribution). */++const policyFor = (sourceKey) => (sourceKey && Object.prototype.hasOwnProperty.call(POLICIES, sourceKey) ? POLICIES[sourceKey] : null);++++/** True when the source's links must be stored and shown exactly as received. */++function keepsLinksVerbatim(sourceKey) {++ const p = policyFor(sourceKey);++ return Boolean(p && p.verbatimLinks);++}++++/** True when the source's content must not reach a browser extension. */++function excludesExtension(sourceKey) {++ const p = policyFor(sourceKey);++ return Boolean(p && p.noExtension);++}++++/** The attribution for a set of source rows ({ source_key, … }): the first whose policy names one. */++function attributionOf(sources) {++ for (const s of sources || []) {++ const p = policyFor(s && s.source_key);++ if (p && p.attribution) return { text: p.attribution.text, url: p.attribution.url };++ }++ return null;++}++++/** True when any of the offer's sources forbids its content in a browser extension. */++function hidesFromExtension(sources) {++ return (sources || []).some((s) => excludesExtension(s && s.source_key));++}++++module.exports = { POLICIES, policyFor, keepsLinksVerbatim, excludesExtension, attributionOf, hidesFromExtension };@@@ -71,6 +71,21 @@ function normalizeUrl(input) {return s.length <= 2048 ? s : null;}++/**++ * An http(s) URL validated but kept exactly as the source stated it — no fragment removal, no++ * tracking-parameter stripping, no host lower-casing, no re-encoding (source policy++ * `verbatimLinks`, e.g. DealNews' feed terms). A URL Deals cannot use is null, as with normalizeUrl.++ */++function verbatimUrl(input) {++ if (input == null || input === '') return null;++ const s = String(input);++ let u;++ try { u = new URL(s); } catch { return null; }++ if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;++ if (u.username || u.password) return null;++ return s.length <= 2048 ? s : null;++}++/** The identity used to find duplicates: normalised URL without scheme and without "www.". */function urlKey(normalized) {if (!normalized) return null;@@@ -79,7 +94,7 @@ function urlKey(normalized) {module.exports = {ApiError, newId, CONDITIONS, AVAILABILITY, AVAILABILITY_LABEL, SCHEMA_AVAILABILITY, SCHEMA_CONDITION,-- normalizeUrl, urlKey,++ normalizeUrl, urlKey, verbatimUrl,domainOf: normalize.domainOf, slugify: normalize.slugify,parseAmount: asApiError(normalize.parseAmount), parseCurrency: asApiError(normalize.parseCurrency),parseEnum: asApiError(normalize.parseEnum), parseInstant: asApiError(normalize.parseInstant),@@@ -32,6 +32,13 @@ const { run, jsonBody, privateNoStore, ApiError } = require('./errors');const { guard } = require('../auth/viewer');const { iso } = require('../domain/values');++// A browser extension calling the API (its Origin is a *-extension:// scheme). DealNews' feed terms++// forbid their feed's content inside a browser extension, so such a caller is served without any++// offer a source policy hides from extensions (server/domain/source-policy.js). This is the only API++// a browser helper reads; Deals exposes no extension-specific route.++const EXTENSION_ORIGIN = /^(chrome|moz|safari-web|safari|ms-browser|devtools)-extension:\/\//i;++const fromExtension = (req) => EXTENSION_ORIGIN.test(String(req.get('origin') || ''));++function cors(origins) {const allowed = new Set(origins);return (req, res, next) => {@@@ -62,6 +69,12 @@ function createApi(ctx) {const opts = (req) => ({ ip: req.viewer.kind === 'service' ? null : req.ip, traceparent: req.ov && req.ov.traceparent });const offerOut = async (offer) => ({ offer: publication.offerDto(await publication.offerView(offer)) });++ // Public reads: an offer a source policy hides from extensions is "not found" for one.++ const readOfferOut = async (req, offer) => {++ const v = await publication.offerView(offer);++ if (fromExtension(req) && v.extensionExcluded) throw new ApiError(404, 'offer.not_found', 'No such offer');++ return { offer: publication.offerDto(v) };++ };const needSubject = (req) => { if (!req.viewer.subject) throw new ApiError(req.viewer.kind === 'service' ? 400 : 401, req.viewer.kind === 'service' ? 'subject.required' : 'auth.required', req.viewer.kind === 'service' ? 'X-OV-Subject must name the person this action is for' : 'Sign in first'); };// ── offers ──────────────────────────────────────────────@@@ -70,7 +83,9 @@ function createApi(ctx) {const page = Math.max(1, parseInt(req.query.page, 10) || 1);const per = 25;const rows = req.query.sort === 'new' ? await listings.newest(per, (page - 1) * per) : await listings.hot(per, (page - 1) * per);-- return { sort: req.query.sort === 'new' ? 'new' : 'hot', page, total: await listings.countActive(), offers: await Promise.all(rows.map(async (o) => publication.offerDto(await publication.offerView(o)))) };++ const views = await Promise.all(rows.map(async (o) => publication.offerView(o)));++ const visible = fromExtension(req) ? views.filter((v) => !v.extensionExcluded) : views;++ return { sort: req.query.sort === 'new' ? 'new' : 'hot', page, total: await listings.countActive(), offers: visible.map((v) => publication.offerDto(v)) };}));// A removed deal answers like its page (410, reason only); moderators still read the whole record.@@@ -80,7 +95,7 @@ function createApi(ctx) {if (root.status === 'disabled' && !access.isModerator(req.viewer)) {throw new ApiError(410, 'offer.disabled', 'This deal was removed by moderators', { id: root.id, slug: root.slug, status: 'disabled', reason: root.disabled_reason });}-- return await offerOut(offer);++ return await readOfferOut(req, offer);}));router.get('/offers/:id/hotness', run(async (req) => {@@@ -162,9 +177,10 @@ function createApi(ctx) {const p = await catalog.productBySlug(req.params.slug) || await catalog.product(req.params.slug);if (!p) throw new ApiError(404, 'product.not_found', 'No such product');const pv = await publication.productView(p);++ const offers = fromExtension(req) ? pv.offers.filter((v) => !v.extensionExcluded) : pv.offers;return {product: { id: p.id, slug: p.slug, name: p.name, brand: p.brand, category: p.category, url: publication.abs(publication.productPath(p)), aliases: pv.aliases.map((a) => ({ kind: a.kind, value: a.value })) },-- offers: pv.offers.map((v) => publication.offerDto(v)),++ offers: offers.map((v) => publication.offerDto(v)),};}));@@@ -185,7 +201,9 @@ function createApi(ctx) {router.get('/stores/:domain', run(async (req) => {const st = await catalog.storeByDomain(req.params.domain);if (!st) throw new ApiError(404, 'store.not_found', 'No such store');-- return { store: { id: st.id, domain: st.domain, name: st.name, url: publication.abs(publication.storePath(st)) }, offers: await Promise.all((await listings.byStore(st.id, 50, 0)).map(async (o) => publication.offerDto(await publication.offerView(o)))) };++ const views = await Promise.all((await listings.byStore(st.id, 50, 0)).map(async (o) => await publication.offerView(o)));++ const offers = fromExtension(req) ? views.filter((v) => !v.extensionExcluded) : views;++ return { store: { id: st.id, domain: st.domain, name: st.name, url: publication.abs(publication.storePath(st)) }, offers: offers.map((v) => publication.offerDto(v)) };}));// ── watches ─────────────────────────────────────────────@@@ -419,9 +419,12 @@ function createPages(ctx) {return Promise.all((await listings.newest(50, 0)).map(async (o) => {const v = await publication.offerView(o);const price = `${views.priceText(v.latest)}${v.latest ? ` as of ${iso(v.latest.observed_at)}` : ''}`;++ // A source policy may require attribution: carried as text in the summary and as a tag.++ const via = v.attribution ? `Deal via ${v.attribution.text}` : null;return {id: `deals:offer:${o.id}`, url: publication.abs(publication.offerPath(o)), title: o.title,-- summary: `${price}${v.store ? ` at ${v.store.name || v.store.domain}` : ''}.${o.description ? ` ${o.description.slice(0, 300)}` : ''}`,++ summary: `${via ? `${via}. ` : ''}${price}${v.store ? ` at ${v.store.name || v.store.domain}` : ''}.${o.description ? ` ${o.description.slice(0, 300)}` : ''}`,++ tags: via ? [via] : undefined,published: o.created_at, updated: Math.max(o.updated_at, v.latest ? v.latest.observed_at : 0), decision: v.decision,};}));@@@ -54,6 +54,16 @@ function statusBadge(root) {const csrfField = (csrf) => html`<input type="hidden" name="csrf" value="${csrf}">`;++/**++ * The attribution line a source policy can require (e.g. DealNews' feed terms): "Deal via <source>"++ * linking to the source's site. Shown wherever the offer's content appears — cards, the offer page++ * and the product comparison. The link to the source never replaces or rewrites the offer's own link.++ */++function attributionLine(a) {++ if (!a) return '';++ return html`<p class="attribution">Deal via <a href="${safeHref(a.url)}" rel="nofollow noopener">${a.text}</a></p>`;++}++/** The one-line "latest observation" block used on cards and pages. */function latestLine(v) {const o = v.latest;@@@ -71,6 +81,7 @@ function offerCard(v, urls) {<div class="deal-body"><h2 class="deal-title"><a href="${urls.offerPath(r)}">${r.title}</a> ${statusBadge(r)}</h2>${latestLine(v)}++${attributionLine(v.attribution)}<p class="meta">${v.store ? html`<a href="${urls.storePath(v.store)}">${v.store.name || v.store.domain}</a> · ` : ''}${v.product ? html`<a href="${urls.productPath(v.product)}">${v.product.name}</a> · ` : ''}posted ${timeEl(r.created_at)}${r.origin === 'import' ? ' · imported from a source' : ''}</p></div></li>`;}@@@ -147,6 +158,7 @@ ${myVote ? html`<button name="value" value="remove">Remove my vote</button>` : '${notice ? html`<p class="notice${notice.error ? ' error' : ''}" role="status">${notice.text}</p>` : ''}<article class="deal" data-offer="${r.id}"><h1>${r.title} ${statusBadge(r)}</h1>++${attributionLine(v.attribution)}${r.status === 'expired' ? html`<p class="notice">This deal expired${r.expired_at ? html` on ${timeEl(r.expired_at)}` : ''}${r.expired_reason === 'stated_expiry' ? ' (its stated end date passed)' : ''}.</p>` : ''}<section class="latest" aria-label="Latest observation">${latestLine(v)}@@@ -207,7 +219,7 @@ ${(mergedComments || []).map((mc) => html`<h3>Discussion from the merged listingfunction productPage({ pv, urls }) {const p = pv.product;const rows = pv.offers.map((v) => html`<tr>--<td><a href="${urls.offerPath(v.root)}">${v.root.title}</a> ${statusBadge(v.root)}</td>++<td><a href="${urls.offerPath(v.root)}">${v.root.title}</a> ${statusBadge(v.root)}${attributionLine(v.attribution)}</td><td>${v.store ? v.store.name || v.store.domain : 'not stated'}</td><td data-price="${v.latest && v.latest.price != null ? v.latest.price : ''}">${priceText(v.latest)}${v.latest && shippingText(v.latest) ? html` <span class="meta">${shippingText(v.latest)}</span>` : ''}</td><td>${v.latest ? timeEl(v.latest.observed_at) : html`<span class="meta">never observed</span>`}</td>@@@ -140,6 +140,45 @@ const { signDelivery, signDeliveryHeaders } = require('openvibe-sdk/events');assert.strictEqual((await post(news, signDeliveryHeaders(news, 'whsec-test'))).json().outcome, 'ignored');});++ await check('a verbatim source keeps its link byte-for-byte and is attributed on the card, page and JSON', async () => {++ // A source policy (source-policy.js, keyed by the Sources source_key) can require its links++ // kept exactly as received and name an attribution — DealNews' feed terms. Everything else++ // keeps today's normalisation (the kettles item above still stripped its utm_source).++ const raw = 'https://www.dealnews.com/deals/acme-espresso/?utm_source=rss&utm_medium=feed&ref=dealnews#top';++ t.sources.put(sourceItem({ id: 'itm_01K5AAAAAAAAAAAAAAAAAAAAA5', sourceKey: 'dealnews-daily', kind: 'article',++ title: 'Acme espresso machine, $199 (DealNews)', url: raw, retrievedAt: t.clock.now() }));++ const s = await t.ctx.importer.pull();++ assert.deepStrictEqual(s.outcomes, { created: 1 }, JSON.stringify(s));++++ const offer = await byUrl(raw);++ assert.ok(offer, 'the link is stored exactly as received');++ assert.strictEqual(offer.url, raw, 'no fragment or utm_/ref parameters stripped');++++ // The card and the page both carry the attribution line and the link as received.++ const escaped = raw.replace(/&/g, '&');++ const page = await t.get(`/d/${offer.slug}`);++ assert.match(page.text, /Deal via <a href="https:\/\/www\.dealnews\.com\/" rel="nofollow noopener">DealNews<\/a>/);++ assert.ok(page.text.includes(`href="${escaped}"`), 'the offer page links to the link as received');++ const card = await t.get('/new');++ assert.match(card.text, /Deal via <a href="https:\/\/www\.dealnews\.com\/" rel="nofollow noopener">DealNews<\/a>/);++++ // ...and the offer's JSON and the feeds.++ const json = await t.offerJson(offer.slug);++ assert.strictEqual(json.link, raw);++ assert.deepStrictEqual(json.attribution, { text: 'DealNews', url: 'https://www.dealnews.com/' });++ assert.match((await t.get('/feed.xml')).text, /Deal via DealNews/);++ assert.match((await t.get('/feed.json')).text, /Deal via DealNews/);++++ // A browser extension is served by the API without it; an ordinary caller still gets it.++ const ext = { headers: { origin: 'chrome-extension://abcdefghijklmnopabcdefghijklmnop' } };++ const list = await t.get('/api/v1/offers?sort=new', ext);++ assert.ok(!list.text.includes('dealnews.com'), 'the extension list omits the verbatim source');++ assert.strictEqual((await t.get(`/api/v1/offers/${offer.id}`, ext)).status, 404);++ const normal = await t.get(`/api/v1/offers/${offer.id}`);++ assert.strictEqual(normal.status, 200);++ assert.deepStrictEqual(normal.json().offer.attribution, { text: 'DealNews', url: 'https://www.dealnews.com/' });++ });++await check('Sources down: the pull fails honestly (readiness says so), keeps its cursor, invents nothing', async () => {const cursor = await t.ctx.importer.cursor();const offers = (await t.ctx.store.db.prepare('SELECT COUNT(*) AS n FROM deal_offers').get()).n;
@@ -30,7 +30,7 @@ The nine charter tables live in Deals' own PostgreSQL database (`ov_deals` on th
@@ -30,7 +30,7 @@ The nine charter tables live in Deals' own PostgreSQL database (`ov_deals` on th
|---|---|
|---|---|
| `deal_products` | product records (name, brand, category), `/p/:slug` |
| `deal_products` | product records (name, brand, category), `/p/:slug` |
| `deal_product_aliases` | `gtin` (normalised to 14 digits), `mpn`, `sku`, exact `name`, `url` → one product; an alias never silently moves to another product |
| `deal_product_aliases` | `gtin` (normalised to 14 digits), `mpn`, `sku`, exact `name`, `url` → one product; an alias never silently moves to another product |
-| `deal_offers` | an offer: title, link (tracking parameters stripped), store, product, status (`active`/`expired`/`disabled`), **stated** expiry, review state, and the merge pointer `merged_into` |
+| `deal_offers` | an offer: title, link (tracking parameters stripped, except where a source policy keeps it verbatim — see "Per-source policy"), store, product, status (`active`/`expired`/`disabled`), **stated** expiry, review state, and the merge pointer `merged_into` |
| `deal_offer_sources` | where an offer came from: the submission, a person's observations, or an OpenVibe.Sources item (typed reference `{sources, item, itm_…, revision}`; the item is never copied) |
| `deal_offer_sources` | where an offer came from: the submission, a person's observations, or an OpenVibe.Sources item (typed reference `{sources, item, itm_…, revision}`; the item is never copied) |
| `deal_votes` | one row per (offer, subject): `+1` / `-1` / `0` (removed, kept as history), with the weight and IP hash it was cast with |
| `deal_votes` | one row per (offer, subject): `+1` / `-1` / `0` (removed, kept as history), with the weight and IP hash it was cast with |
| `deal_hotness_snapshots` | every hotness computation with all of its inputs (formula `hot@1`) |
| `deal_hotness_snapshots` | every hotness computation with all of its inputs (formula `hot@1`) |
@@ -135,7 +135,8 @@ Consumed: `sources.item.created|updated|removed` at `POST /internal/events` (sig
@@ -135,7 +135,8 @@ Consumed: `sources.item.created|updated|removed` at `POST /internal/events` (sig
- The same link (normalised: fragment and tracking/affiliate parameters removed, `www.` and scheme
- The same link (normalised: fragment and tracking/affiliate parameters removed, `www.` and scheme
ignored) cannot be posted twice: a person gets the existing deal (409 / redirect), a second Sources
ignored) cannot be posted twice: a person gets the existing deal (409 / redirect), a second Sources
- item attaches to it as another source.
+ item attaches to it as another source. A link a source policy keeps verbatim is identified as
+ received, so it never strips parameters to collide with a normalised one.
- Moderators merge listings of one deal (`/mod`, "possible duplicates" = same store and same product
- Moderators merge listings of one deal (`/mod`, "possible duplicates" = same store and same product
or title). **Merging moves nothing:** B gets `merged_into = A`; A's page, JSON, votes, history and
or title). **Merging moves nothing:** B gets `merged_into = A`; A's page, JSON, votes, history and
sources read across the group; B's URL answers 301. A person who voted on both counts **once**
sources read across the group; B's URL answers 301. A person who voted on both counts **once**
@@ -227,6 +228,37 @@ when Sources fetched them again. A removed item (takedown, licence) removes its
@@ -227,6 +228,37 @@ when Sources fetched them again. A removed item (takedown, licence) removes its
offer with no source left is disabled. Imported text is `review_state = pending` → `noindex` until a
offer with no source left is disabled. Imported text is `review_state = pending` → `noindex` until a
moderator records a review (a `usr_` subject; a service cannot review).
moderator records a review (a `usr_` subject; a service cannot review).
+### Per-source policy: verbatim links, attribution and browser extensions
+
+Deals normalises an imported offer's link by default: the fragment and tracking/affiliate parameters
+are removed so two posts of one deal share an identity. A source whose terms require otherwise gets
+an explicit entry in `server/domain/source-policy.js`, keyed by the Sources `source_key` — the prose
+`license_note` / `terms_note` are never parsed, the map is data:
+
+| Policy field | Effect |
+|---|---|
+| `verbatimLinks` | the link is stored and shown exactly as received (no fragment or tracking-parameter stripping, no rewriting, no `rel` change that alters the URL), and the title is used as the source stated it, with no added words |
+| `attribution` | `"Deal via <text>"` linking to `url` is shown on the offer card, the offer page and the product comparison, and carried in the offer's JSON (`.json` / API `attribution`) and its RSS/Atom/JSON-Feed summary and tags |
+| `noExtension` | the offer is excluded from the API when the caller's `Origin` is a browser-extension scheme (`chrome-extension://`, `moz-extension://`, …): absent from `GET /offers`, `GET /products/:slug` and `GET /stores/:domain`, and `GET /offers/:id` answers 404 |
+
+**DealNews** ([feed terms](https://www.dealnews.com/pages/rss.html), checked 2026-10-07) allows public
+use of its feeds only without removing or adding content within the feed display, without modifying
+the links or referral codes, with attribution to DealNews as text (e.g. "DealNews") or their 88x31
+logo, and never inside a browser extension. `dealnews-daily` therefore sets all three:
+
+```js
+'dealnews-daily': {
+ verbatimLinks: true,
+ attribution: { text: 'DealNews', url: 'https://www.dealnews.com/' },
+ noExtension: true,
+},
+```
+
+Community submissions and every source with no entry keep today's behaviour (normalised links, no
+attribution, visible to every caller). `test/import.test.js` and `test/api.test.js` cover the verbatim
+link byte-for-byte, the attribution on card, page and JSON, the untouched default, and the
+extension-facing exclusion.
+
## Discoverability (roadmap §32)
## Discoverability (roadmap §32)
- SSR HTML with canonical, Open Graph, JSON-LD (`Product`/`Offer`, `BreadcrumbList`, `WebSite` with
- SSR HTML with canonical, Open Graph, JSON-LD (`Product`/`Offer`, `BreadcrumbList`, `WebSite` with
@@ -262,7 +294,8 @@ render time, so the cache stays short); signed-in views, forms, API responses, e
@@ -262,7 +294,8 @@ render time, so the cache stays short); signed-in views, forms, API responses, e
- **OpenVibe.Network:** SSO (OAuth client `deals`, registered in production), JWKS,
- **OpenVibe.Network:** SSO (OAuth client `deals`, registered in production), JWKS,
`identity.subject.resolve`.
`identity.subject.resolve`.
- **OpenVibe.Sources:** `sources.item.read`; a `deals`-category source must be registered and enabled
- **OpenVibe.Sources:** `sources.item.read`; a `deals`-category source must be registered and enabled
- there (the seeded `dealnews-daily` is disabled until a person re-verifies its terms).
+ there (the seeded `dealnews-daily` is disabled until a person re-verifies its terms; its feed terms
+ are now encoded in `server/domain/source-policy.js`, so enabling it stays within them).
- **OpenVibe.Community:** `community.comment.write`, optionally `community.comment.moderate`.
- **OpenVibe.Community:** `community.comment.write`, optionally `community.comment.moderate`.
- **OpenVibe.Events:** `events.event.publish`; `events.subscription.manage` once, for
- **OpenVibe.Events:** `events.event.publish`; `events.subscription.manage` once, for
`scripts/subscribe.js`.
`scripts/subscribe.js`.
@@ -20,6 +20,7 @@
@@ -20,6 +20,7 @@
*/
*/
const seo = require('openvibe-publishing/seo');
const seo = require('openvibe-publishing/seo');
const { iso } = require('./values');
const { iso } = require('./values');
+const { attributionOf, hidesFromExtension } = require('./source-policy');
function createPublication({ config, store, reads, catalog }) {
function createPublication({ config, store, reads, catalog }) {
const abs = (p) => seo.canonicalUrl(config.baseUrl, p);
const abs = (p) => seo.canonicalUrl(config.baseUrl, p);
@@ -65,6 +66,7 @@ function createPublication({ config, store, reads, catalog }) {
@@ -65,6 +66,7 @@ function createPublication({ config, store, reads, catalog }) {
const latest = await reads.latestObservation(ids);
const latest = await reads.latestObservation(ids);
const tally = await reads.tally(ids);
const tally = await reads.tally(ids);
const snap = await reads.lastSnapshot(root.id);
const snap = await reads.lastSnapshot(root.id);
+ const sources = await reads.sources(ids);
return {
return {
root,
root,
ids,
ids,
@@ -73,7 +75,11 @@ function createPublication({ config, store, reads, catalog }) {
@@ -73,7 +75,11 @@ function createPublication({ config, store, reads, catalog }) {
observations: await reads.observations(ids),
observations: await reads.observations(ids),
latest,
latest,
freshness: freshness(latest, now),
freshness: freshness(latest, now),
- sources: await reads.sources(ids),
+ sources,
+ // A source policy may require attribution (shown on cards, the page, JSON and feeds) and
+ // may forbid the offer's content in a browser extension (DealNews' feed terms).
+ attribution: attributionOf(sources),
+ extensionExcluded: hidesFromExtension(sources),
members: await Promise.all(ids.filter((id) => id !== root.id).map(async (id) => await reads.get(id))),
members: await Promise.all(ids.filter((id) => id !== root.id).map(async (id) => await reads.get(id))),
tally,
tally,
hotness: snap,
hotness: snap,
@@ -107,6 +113,7 @@ function createPublication({ config, store, reads, catalog }) {
@@ -107,6 +113,7 @@ function createPublication({ config, store, reads, catalog }) {
// "latest", never "current": whether it still holds is what freshness says.
// "latest", never "current": whether it still holds is what freshness says.
latest_observation: v.latest ? observationDto(v.latest) : null,
latest_observation: v.latest ? observationDto(v.latest) : null,
freshness: v.freshness,
freshness: v.freshness,
+ attribution: v.attribution || null,
observations: v.observations.map(observationDto),
observations: v.observations.map(observationDto),
sources: v.sources.map((s) => ({
sources: v.sources.map((s) => ({
id: s.id, kind: s.kind, listing: s.offer_id,
id: s.id, kind: s.kind, listing: s.offer_id,
@@ -22,6 +22,10 @@
@@ -22,6 +22,10 @@
* headline — it stays "not stated" until a structured source or a person states it
* headline — it stays "not stated" until a structured source or a person states it
* other kinds skipped
* other kinds skipped
*
*
+ * A source policy (server/domain/source-policy.js) may require a source's links kept verbatim and
+ * name an attribution (DealNews' feed terms); such a source's links are stored and shown exactly as
+ * received, its title is used as stated, and the attribution is shown with every offer it touches.
+ *
* observed_at = the item's provenance.retrieved_at (when Sources saw it). An item Deals already
* observed_at = the item's provenance.retrieved_at (when Sources saw it). An item Deals already
* imported produces a new observation only when its revision or its retrieved_at advanced, so
* imported produces a new observation only when its revision or its retrieved_at advanced, so
* replays are no-ops. Two items (or a submission and an item) with the same link attach to the same
* replays are no-ops. Two items (or a submission and an item) with the same link attach to the same
@@ -33,8 +37,9 @@
@@ -33,8 +37,9 @@
*/
*/
const { createChangeCursor, pullChanges } = require('openvibe-publishing/ingest');
const { createChangeCursor, pullChanges } = require('openvibe-publishing/ingest');
const {
const {
- normalizeUrl, urlKey, parseAmount, SCHEMA_AVAILABILITY, SCHEMA_CONDITION, iso,
+ normalizeUrl, verbatimUrl, urlKey, parseAmount, SCHEMA_AVAILABILITY, SCHEMA_CONDITION, iso,
} = require('./values');
} = require('./values');
+const { keepsLinksVerbatim } = require('./source-policy');
const CURSOR = 'sources.deals.after';
const CURSOR = 'sources.deals.after';
@@ -88,8 +93,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
@@ -88,8 +93,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
function candidates(item) {
function candidates(item) {
const f = item.fields || {};
const f = item.fields || {};
const stated = (t) => { const ms = t ? Date.parse(t) : NaN; return Number.isFinite(ms) ? ms : null; };
const stated = (t) => { const ms = t ? Date.parse(t) : NaN; return Number.isFinite(ms) ? ms : null; };
+ // A source policy may require its links to be kept exactly as received (DealNews' feed
+ // terms); every other source keeps today's normalisation.
+ const verbatim = keepsLinksVerbatim(item.source_key);
+ const linkOf = (raw) => (verbatim ? verbatimUrl(raw) : normalizeUrl(raw));
if (item.kind === 'offer') {
if (item.kind === 'offer') {
- const url = normalizeUrl(f.url || item.canonical_url);
+ const url = linkOf(f.url || item.canonical_url);
if (!url || !item.title) return [];
if (!url || !item.title) return [];
return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields(f), expires_at: stated(f.valid_until), store_name: f.seller || null }];
return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields(f), expires_at: stated(f.valid_until), store_name: f.seller || null }];
}
}
@@ -98,11 +107,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
@@ -98,11 +107,12 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
const seen = new Set();
const seen = new Set();
const out = [];
const out = [];
for (const o of (Array.isArray(f.offers) ? f.offers : []).slice(0, 20)) {
for (const o of (Array.isArray(f.offers) ? f.offers : []).slice(0, 20)) {
- const url = normalizeUrl(o.url || item.canonical_url);
+ const url = linkOf(o.url || item.canonical_url);
if (!url || seen.has(urlKey(url))) continue;
if (!url || seen.has(urlKey(url))) continue;
seen.add(urlKey(url));
seen.add(urlKey(url));
out.push({
out.push({
- part: urlKey(url), url, title: o.seller ? `${item.title} — ${o.seller}` : item.title, description: item.summary,
+ // A verbatim source's title is used as stated, with no added words.
+ part: urlKey(url), url, title: !verbatim && o.seller ? `${item.title} — ${o.seller}` : item.title, description: item.summary,
obs: obsFields(o), expires_at: stated(o.valid_until), store_name: o.seller || null,
obs: obsFields(o), expires_at: stated(o.valid_until), store_name: o.seller || null,
product: { name: item.title, brand: f.brand, gtin: f.gtin, mpn: f.mpn, sku: f.sku },
product: { name: item.title, brand: f.brand, gtin: f.gtin, mpn: f.mpn, sku: f.sku },
});
});
@@ -110,7 +120,7 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
@@ -110,7 +120,7 @@ function createImporter({ config, store, reads, catalog, offers, indexing, sourc
return out;
return out;
}
}
if (item.kind === 'article' || item.kind === 'record') {
if (item.kind === 'article' || item.kind === 'record') {
- const url = normalizeUrl(item.canonical_url);
+ const url = linkOf(item.canonical_url);
if (!url || !item.title) return [];
if (!url || !item.title) return [];
return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields({}), expires_at: null, store_name: null }];
return [{ part: '', url, title: item.title, description: item.summary, obs: obsFields({}), expires_at: null, store_name: null }];
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,57 @@
@@ -0,0 +1,57 @@
+'use strict';
+
+/**
+ * Per-source policy for imported offers: which links must be kept exactly as the source stated them,
+ * how the source is attributed, and whether its content may reach a browser extension.
+ *
+ * A source that hands Deals a feed with terms gets an entry here, explicitly, keyed by the
+ * OpenVibe.Sources `source_key` (the key Sources labels each item with, e.g. `dealnews-daily`).
+ * Sources' `license_note` / `terms_note` are prose and are never parsed: a source with no entry gets
+ * today's default — links normalised (tracking and affiliate parameters stripped) and no attribution.
+ *
+ * DealNews' feed terms (https://www.dealnews.com/pages/rss.html) allow public use of their feeds only
+ * (1) without removing or adding content within the feed display, (2) without modifying the links or
+ * referral codes, (3) with attribution to DealNews as text (e.g. "DealNews") or with their 88x31 logo,
+ * and (4) never inside a browser extension. `dealnews-daily` therefore keeps its links verbatim, is
+ * attributed wherever the offer appears, and is kept out of the API a browser extension could read.
+ */
+
+const POLICIES = Object.freeze({
+ 'dealnews-daily': Object.freeze({
+ verbatimLinks: true,
+ attribution: Object.freeze({ text: 'DealNews', url: 'https://www.dealnews.com/' }),
+ // Terms (4): the feed may never be used inside a browser extension.
+ noExtension: true,
+ }),
+});
+
+/** The policy for a Sources `source_key`, or null (today's default: normalise the link, no attribution). */
+const policyFor = (sourceKey) => (sourceKey && Object.prototype.hasOwnProperty.call(POLICIES, sourceKey) ? POLICIES[sourceKey] : null);
+
+/** True when the source's links must be stored and shown exactly as received. */
+function keepsLinksVerbatim(sourceKey) {
+ const p = policyFor(sourceKey);
+ return Boolean(p && p.verbatimLinks);
+}
+
+/** True when the source's content must not reach a browser extension. */
+function excludesExtension(sourceKey) {
+ const p = policyFor(sourceKey);
+ return Boolean(p && p.noExtension);
+}
+
+/** The attribution for a set of source rows ({ source_key, … }): the first whose policy names one. */
+function attributionOf(sources) {
+ for (const s of sources || []) {
+ const p = policyFor(s && s.source_key);
+ if (p && p.attribution) return { text: p.attribution.text, url: p.attribution.url };
+ }
+ return null;
+}
+
+/** True when any of the offer's sources forbids its content in a browser extension. */
+function hidesFromExtension(sources) {
+ return (sources || []).some((s) => excludesExtension(s && s.source_key));
+}
+
+module.exports = { POLICIES, policyFor, keepsLinksVerbatim, excludesExtension, attributionOf, hidesFromExtension };
@@ -71,6 +71,21 @@ function normalizeUrl(input) {
@@ -71,6 +71,21 @@ function normalizeUrl(input) {
return s.length <= 2048 ? s : null;
return s.length <= 2048 ? s : null;
}
}
+/**
+ * An http(s) URL validated but kept exactly as the source stated it — no fragment removal, no
+ * tracking-parameter stripping, no host lower-casing, no re-encoding (source policy
+ * `verbatimLinks`, e.g. DealNews' feed terms). A URL Deals cannot use is null, as with normalizeUrl.
+ */
+function verbatimUrl(input) {
+ if (input == null || input === '') return null;
+ const s = String(input);
+ let u;
+ try { u = new URL(s); } catch { return null; }
+ if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;
+ if (u.username || u.password) return null;
+ return s.length <= 2048 ? s : null;
+}
+
/** The identity used to find duplicates: normalised URL without scheme and without "www.". */
/** The identity used to find duplicates: normalised URL without scheme and without "www.". */
function urlKey(normalized) {
function urlKey(normalized) {
if (!normalized) return null;
if (!normalized) return null;
@@ -79,7 +94,7 @@ function urlKey(normalized) {
@@ -79,7 +94,7 @@ function urlKey(normalized) {
module.exports = {
module.exports = {
ApiError, newId, CONDITIONS, AVAILABILITY, AVAILABILITY_LABEL, SCHEMA_AVAILABILITY, SCHEMA_CONDITION,
ApiError, newId, CONDITIONS, AVAILABILITY, AVAILABILITY_LABEL, SCHEMA_AVAILABILITY, SCHEMA_CONDITION,
- normalizeUrl, urlKey,
+ normalizeUrl, urlKey, verbatimUrl,
domainOf: normalize.domainOf, slugify: normalize.slugify,
domainOf: normalize.domainOf, slugify: normalize.slugify,
parseAmount: asApiError(normalize.parseAmount), parseCurrency: asApiError(normalize.parseCurrency),
parseAmount: asApiError(normalize.parseAmount), parseCurrency: asApiError(normalize.parseCurrency),
parseEnum: asApiError(normalize.parseEnum), parseInstant: asApiError(normalize.parseInstant),
parseEnum: asApiError(normalize.parseEnum), parseInstant: asApiError(normalize.parseInstant),
@@ -32,6 +32,13 @@ const { run, jsonBody, privateNoStore, ApiError } = require('./errors');
@@ -32,6 +32,13 @@ const { run, jsonBody, privateNoStore, ApiError } = require('./errors');
const { guard } = require('../auth/viewer');
const { guard } = require('../auth/viewer');
const { iso } = require('../domain/values');
const { iso } = require('../domain/values');
+// A browser extension calling the API (its Origin is a *-extension:// scheme). DealNews' feed terms
+// forbid their feed's content inside a browser extension, so such a caller is served without any
+// offer a source policy hides from extensions (server/domain/source-policy.js). This is the only API
+// a browser helper reads; Deals exposes no extension-specific route.
+const EXTENSION_ORIGIN = /^(chrome|moz|safari-web|safari|ms-browser|devtools)-extension:\/\//i;
+const fromExtension = (req) => EXTENSION_ORIGIN.test(String(req.get('origin') || ''));
+
function cors(origins) {
function cors(origins) {
const allowed = new Set(origins);
const allowed = new Set(origins);
return (req, res, next) => {
return (req, res, next) => {
@@ -62,6 +69,12 @@ function createApi(ctx) {
@@ -62,6 +69,12 @@ function createApi(ctx) {
const opts = (req) => ({ ip: req.viewer.kind === 'service' ? null : req.ip, traceparent: req.ov && req.ov.traceparent });
const opts = (req) => ({ ip: req.viewer.kind === 'service' ? null : req.ip, traceparent: req.ov && req.ov.traceparent });
const offerOut = async (offer) => ({ offer: publication.offerDto(await publication.offerView(offer)) });
const offerOut = async (offer) => ({ offer: publication.offerDto(await publication.offerView(offer)) });
+ // Public reads: an offer a source policy hides from extensions is "not found" for one.
+ const readOfferOut = async (req, offer) => {
+ const v = await publication.offerView(offer);
+ if (fromExtension(req) && v.extensionExcluded) throw new ApiError(404, 'offer.not_found', 'No such offer');
+ return { offer: publication.offerDto(v) };
+ };
const needSubject = (req) => { if (!req.viewer.subject) throw new ApiError(req.viewer.kind === 'service' ? 400 : 401, req.viewer.kind === 'service' ? 'subject.required' : 'auth.required', req.viewer.kind === 'service' ? 'X-OV-Subject must name the person this action is for' : 'Sign in first'); };
const needSubject = (req) => { if (!req.viewer.subject) throw new ApiError(req.viewer.kind === 'service' ? 400 : 401, req.viewer.kind === 'service' ? 'subject.required' : 'auth.required', req.viewer.kind === 'service' ? 'X-OV-Subject must name the person this action is for' : 'Sign in first'); };
// ── offers ──────────────────────────────────────────────
// ── offers ──────────────────────────────────────────────
@@ -70,7 +83,9 @@ function createApi(ctx) {
@@ -70,7 +83,9 @@ function createApi(ctx) {
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
const page = Math.max(1, parseInt(req.query.page, 10) || 1);
const per = 25;
const per = 25;
const rows = req.query.sort === 'new' ? await listings.newest(per, (page - 1) * per) : await listings.hot(per, (page - 1) * per);
const rows = req.query.sort === 'new' ? await listings.newest(per, (page - 1) * per) : await listings.hot(per, (page - 1) * per);
- return { sort: req.query.sort === 'new' ? 'new' : 'hot', page, total: await listings.countActive(), offers: await Promise.all(rows.map(async (o) => publication.offerDto(await publication.offerView(o)))) };
+ const views = await Promise.all(rows.map(async (o) => publication.offerView(o)));
+ const visible = fromExtension(req) ? views.filter((v) => !v.extensionExcluded) : views;
+ return { sort: req.query.sort === 'new' ? 'new' : 'hot', page, total: await listings.countActive(), offers: visible.map((v) => publication.offerDto(v)) };
}));
}));
// A removed deal answers like its page (410, reason only); moderators still read the whole record.
// A removed deal answers like its page (410, reason only); moderators still read the whole record.
@@ -80,7 +95,7 @@ function createApi(ctx) {
@@ -80,7 +95,7 @@ function createApi(ctx) {
if (root.status === 'disabled' && !access.isModerator(req.viewer)) {
if (root.status === 'disabled' && !access.isModerator(req.viewer)) {
throw new ApiError(410, 'offer.disabled', 'This deal was removed by moderators', { id: root.id, slug: root.slug, status: 'disabled', reason: root.disabled_reason });
throw new ApiError(410, 'offer.disabled', 'This deal was removed by moderators', { id: root.id, slug: root.slug, status: 'disabled', reason: root.disabled_reason });
}
}
- return await offerOut(offer);
+ return await readOfferOut(req, offer);
}));
}));
router.get('/offers/:id/hotness', run(async (req) => {
router.get('/offers/:id/hotness', run(async (req) => {
@@ -162,9 +177,10 @@ function createApi(ctx) {
@@ -162,9 +177,10 @@ function createApi(ctx) {
const p = await catalog.productBySlug(req.params.slug) || await catalog.product(req.params.slug);
const p = await catalog.productBySlug(req.params.slug) || await catalog.product(req.params.slug);
if (!p) throw new ApiError(404, 'product.not_found', 'No such product');
if (!p) throw new ApiError(404, 'product.not_found', 'No such product');
const pv = await publication.productView(p);
const pv = await publication.productView(p);
+ const offers = fromExtension(req) ? pv.offers.filter((v) => !v.extensionExcluded) : pv.offers;
return {
return {
product: { id: p.id, slug: p.slug, name: p.name, brand: p.brand, category: p.category, url: publication.abs(publication.productPath(p)), aliases: pv.aliases.map((a) => ({ kind: a.kind, value: a.value })) },
product: { id: p.id, slug: p.slug, name: p.name, brand: p.brand, category: p.category, url: publication.abs(publication.productPath(p)), aliases: pv.aliases.map((a) => ({ kind: a.kind, value: a.value })) },
- offers: pv.offers.map((v) => publication.offerDto(v)),
+ offers: offers.map((v) => publication.offerDto(v)),
};
};
}));
}));
@@ -185,7 +201,9 @@ function createApi(ctx) {
@@ -185,7 +201,9 @@ function createApi(ctx) {
router.get('/stores/:domain', run(async (req) => {
router.get('/stores/:domain', run(async (req) => {
const st = await catalog.storeByDomain(req.params.domain);
const st = await catalog.storeByDomain(req.params.domain);
if (!st) throw new ApiError(404, 'store.not_found', 'No such store');
if (!st) throw new ApiError(404, 'store.not_found', 'No such store');
- return { store: { id: st.id, domain: st.domain, name: st.name, url: publication.abs(publication.storePath(st)) }, offers: await Promise.all((await listings.byStore(st.id, 50, 0)).map(async (o) => publication.offerDto(await publication.offerView(o)))) };
+ const views = await Promise.all((await listings.byStore(st.id, 50, 0)).map(async (o) => await publication.offerView(o)));
+ const offers = fromExtension(req) ? views.filter((v) => !v.extensionExcluded) : views;
+ return { store: { id: st.id, domain: st.domain, name: st.name, url: publication.abs(publication.storePath(st)) }, offers: offers.map((v) => publication.offerDto(v)) };
}));
}));
// ── watches ─────────────────────────────────────────────
// ── watches ─────────────────────────────────────────────
@@ -419,9 +419,12 @@ function createPages(ctx) {
@@ -419,9 +419,12 @@ function createPages(ctx) {
return Promise.all((await listings.newest(50, 0)).map(async (o) => {
return Promise.all((await listings.newest(50, 0)).map(async (o) => {
const v = await publication.offerView(o);
const v = await publication.offerView(o);
const price = `${views.priceText(v.latest)}${v.latest ? ` as of ${iso(v.latest.observed_at)}` : ''}`;
const price = `${views.priceText(v.latest)}${v.latest ? ` as of ${iso(v.latest.observed_at)}` : ''}`;
+ // A source policy may require attribution: carried as text in the summary and as a tag.
+ const via = v.attribution ? `Deal via ${v.attribution.text}` : null;
return {
return {
id: `deals:offer:${o.id}`, url: publication.abs(publication.offerPath(o)), title: o.title,
id: `deals:offer:${o.id}`, url: publication.abs(publication.offerPath(o)), title: o.title,
- summary: `${price}${v.store ? ` at ${v.store.name || v.store.domain}` : ''}.${o.description ? ` ${o.description.slice(0, 300)}` : ''}`,
+ summary: `${via ? `${via}. ` : ''}${price}${v.store ? ` at ${v.store.name || v.store.domain}` : ''}.${o.description ? ` ${o.description.slice(0, 300)}` : ''}`,
+ tags: via ? [via] : undefined,
published: o.created_at, updated: Math.max(o.updated_at, v.latest ? v.latest.observed_at : 0), decision: v.decision,
published: o.created_at, updated: Math.max(o.updated_at, v.latest ? v.latest.observed_at : 0), decision: v.decision,
};
};
}));
}));
@@ -54,6 +54,16 @@ function statusBadge(root) {
@@ -54,6 +54,16 @@ function statusBadge(root) {
const csrfField = (csrf) => html`<input type="hidden" name="csrf" value="${csrf}">`;
const csrfField = (csrf) => html`<input type="hidden" name="csrf" value="${csrf}">`;
+/**
+ * The attribution line a source policy can require (e.g. DealNews' feed terms): "Deal via <source>"
+ * linking to the source's site. Shown wherever the offer's content appears — cards, the offer page
+ * and the product comparison. The link to the source never replaces or rewrites the offer's own link.
+ */
+function attributionLine(a) {
+ if (!a) return '';
+ return html`<p class="attribution">Deal via <a href="${safeHref(a.url)}" rel="nofollow noopener">${a.text}</a></p>`;
+}
+
/** The one-line "latest observation" block used on cards and pages. */
/** The one-line "latest observation" block used on cards and pages. */
function latestLine(v) {
function latestLine(v) {
const o = v.latest;
const o = v.latest;
@@ -71,6 +81,7 @@ function offerCard(v, urls) {
@@ -71,6 +81,7 @@ function offerCard(v, urls) {
<div class="deal-body">
<div class="deal-body">
<h2 class="deal-title"><a href="${urls.offerPath(r)}">${r.title}</a> ${statusBadge(r)}</h2>
<h2 class="deal-title"><a href="${urls.offerPath(r)}">${r.title}</a> ${statusBadge(r)}</h2>
${latestLine(v)}
${latestLine(v)}
+${attributionLine(v.attribution)}
<p class="meta">${v.store ? html`<a href="${urls.storePath(v.store)}">${v.store.name || v.store.domain}</a> · ` : ''}${v.product ? html`<a href="${urls.productPath(v.product)}">${v.product.name}</a> · ` : ''}posted ${timeEl(r.created_at)}${r.origin === 'import' ? ' · imported from a source' : ''}</p>
<p class="meta">${v.store ? html`<a href="${urls.storePath(v.store)}">${v.store.name || v.store.domain}</a> · ` : ''}${v.product ? html`<a href="${urls.productPath(v.product)}">${v.product.name}</a> · ` : ''}posted ${timeEl(r.created_at)}${r.origin === 'import' ? ' · imported from a source' : ''}</p>
</div></li>`;
</div></li>`;
}
}
@@ -147,6 +158,7 @@ ${myVote ? html`<button name="value" value="remove">Remove my vote</button>` : '
@@ -147,6 +158,7 @@ ${myVote ? html`<button name="value" value="remove">Remove my vote</button>` : '
${notice ? html`<p class="notice${notice.error ? ' error' : ''}" role="status">${notice.text}</p>` : ''}
${notice ? html`<p class="notice${notice.error ? ' error' : ''}" role="status">${notice.text}</p>` : ''}
<article class="deal" data-offer="${r.id}">
<article class="deal" data-offer="${r.id}">
<h1>${r.title} ${statusBadge(r)}</h1>
<h1>${r.title} ${statusBadge(r)}</h1>
+${attributionLine(v.attribution)}
${r.status === 'expired' ? html`<p class="notice">This deal expired${r.expired_at ? html` on ${timeEl(r.expired_at)}` : ''}${r.expired_reason === 'stated_expiry' ? ' (its stated end date passed)' : ''}.</p>` : ''}
${r.status === 'expired' ? html`<p class="notice">This deal expired${r.expired_at ? html` on ${timeEl(r.expired_at)}` : ''}${r.expired_reason === 'stated_expiry' ? ' (its stated end date passed)' : ''}.</p>` : ''}
<section class="latest" aria-label="Latest observation">
<section class="latest" aria-label="Latest observation">
${latestLine(v)}
${latestLine(v)}
@@ -207,7 +219,7 @@ ${(mergedComments || []).map((mc) => html`<h3>Discussion from the merged listing
@@ -207,7 +219,7 @@ ${(mergedComments || []).map((mc) => html`<h3>Discussion from the merged listing
function productPage({ pv, urls }) {
function productPage({ pv, urls }) {
const p = pv.product;
const p = pv.product;
const rows = pv.offers.map((v) => html`<tr>
const rows = pv.offers.map((v) => html`<tr>
-<td><a href="${urls.offerPath(v.root)}">${v.root.title}</a> ${statusBadge(v.root)}</td>
+<td><a href="${urls.offerPath(v.root)}">${v.root.title}</a> ${statusBadge(v.root)}${attributionLine(v.attribution)}</td>
<td>${v.store ? v.store.name || v.store.domain : 'not stated'}</td>
<td>${v.store ? v.store.name || v.store.domain : 'not stated'}</td>
<td data-price="${v.latest && v.latest.price != null ? v.latest.price : ''}">${priceText(v.latest)}${v.latest && shippingText(v.latest) ? html` <span class="meta">${shippingText(v.latest)}</span>` : ''}</td>
<td data-price="${v.latest && v.latest.price != null ? v.latest.price : ''}">${priceText(v.latest)}${v.latest && shippingText(v.latest) ? html` <span class="meta">${shippingText(v.latest)}</span>` : ''}</td>
<td>${v.latest ? timeEl(v.latest.observed_at) : html`<span class="meta">never observed</span>`}</td>
<td>${v.latest ? timeEl(v.latest.observed_at) : html`<span class="meta">never observed</span>`}</td>
@@ -140,6 +140,45 @@ const { signDelivery, signDeliveryHeaders } = require('openvibe-sdk/events');
@@ -140,6 +140,45 @@ const { signDelivery, signDeliveryHeaders } = require('openvibe-sdk/events');
assert.strictEqual((await post(news, signDeliveryHeaders(news, 'whsec-test'))).json().outcome, 'ignored');
assert.strictEqual((await post(news, signDeliveryHeaders(news, 'whsec-test'))).json().outcome, 'ignored');
});
});
+ await check('a verbatim source keeps its link byte-for-byte and is attributed on the card, page and JSON', async () => {
+ // A source policy (source-policy.js, keyed by the Sources source_key) can require its links
+ // kept exactly as received and name an attribution — DealNews' feed terms. Everything else
+ // keeps today's normalisation (the kettles item above still stripped its utm_source).
+ const raw = 'https://www.dealnews.com/deals/acme-espresso/?utm_source=rss&utm_medium=feed&ref=dealnews#top';
+ t.sources.put(sourceItem({ id: 'itm_01K5AAAAAAAAAAAAAAAAAAAAA5', sourceKey: 'dealnews-daily', kind: 'article',
+ title: 'Acme espresso machine, $199 (DealNews)', url: raw, retrievedAt: t.clock.now() }));
+ const s = await t.ctx.importer.pull();
+ assert.deepStrictEqual(s.outcomes, { created: 1 }, JSON.stringify(s));
+
+ const offer = await byUrl(raw);
+ assert.ok(offer, 'the link is stored exactly as received');
+ assert.strictEqual(offer.url, raw, 'no fragment or utm_/ref parameters stripped');
+
+ // The card and the page both carry the attribution line and the link as received.
+ const escaped = raw.replace(/&/g, '&');
+ const page = await t.get(`/d/${offer.slug}`);
+ assert.match(page.text, /Deal via <a href="https:\/\/www\.dealnews\.com\/" rel="nofollow noopener">DealNews<\/a>/);
+ assert.ok(page.text.includes(`href="${escaped}"`), 'the offer page links to the link as received');
+ const card = await t.get('/new');
+ assert.match(card.text, /Deal via <a href="https:\/\/www\.dealnews\.com\/" rel="nofollow noopener">DealNews<\/a>/);
+
+ // ...and the offer's JSON and the feeds.
+ const json = await t.offerJson(offer.slug);
+ assert.strictEqual(json.link, raw);
+ assert.deepStrictEqual(json.attribution, { text: 'DealNews', url: 'https://www.dealnews.com/' });
+ assert.match((await t.get('/feed.xml')).text, /Deal via DealNews/);
+ assert.match((await t.get('/feed.json')).text, /Deal via DealNews/);
+
+ // A browser extension is served by the API without it; an ordinary caller still gets it.
+ const ext = { headers: { origin: 'chrome-extension://abcdefghijklmnopabcdefghijklmnop' } };
+ const list = await t.get('/api/v1/offers?sort=new', ext);
+ assert.ok(!list.text.includes('dealnews.com'), 'the extension list omits the verbatim source');
+ assert.strictEqual((await t.get(`/api/v1/offers/${offer.id}`, ext)).status, 404);
+ const normal = await t.get(`/api/v1/offers/${offer.id}`);
+ assert.strictEqual(normal.status, 200);
+ assert.deepStrictEqual(normal.json().offer.attribution, { text: 'DealNews', url: 'https://www.dealnews.com/' });
+ });
+
await check('Sources down: the pull fails honestly (readiness says so), keeps its cursor, invents nothing', async () => {
await check('Sources down: the pull fails honestly (readiness says so), keeps its cursor, invents nothing', async () => {
const cursor = await t.ctx.importer.cursor();
const cursor = await t.ctx.importer.cursor();
const offers = (await t.ctx.store.db.prepare('SELECT COUNT(*) AS n FROM deal_offers').get()).n;
const offers = (await t.ctx.store.db.prepare('SELECT COUNT(*) AS n FROM deal_offers').get()).n;