openvibers/openvibe.food Commits
A food's page shows USDA ERS's annual fruit and vegetable price for 16 items beside BLS (#211)
Changes
@@@ -167,6 +167,15 @@ published average-price file once a day (source `bls-average-prices`) and Food f`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list'sown band stays as it is.++**USDA prices** — the U.S. Department of Agriculture's Economic Research Service publishes annual national-average++retail prices for 155 fresh and processed fruits and vegetables (Fruit and Vegetable Prices, 2023 retail scanner data,++public domain); 16 of Food's items are among them, and an item's page shows ERS's figure for the food's pack beside++BLS's where both price it ([server/food/usda-prices.js](server/food/usda-prices.js)). Food never calls ERS:++OpenVibe.Sources reads the two published CSV tables once a week (source `usda-ers-fruit-vegetable-prices`) and Food++follows its change feed into `food_usda_prices`. The page gives the data year (2023) and repeats ERS's warning that the++data should not be used for making inferences about price changes over time; the annual figure is never shown as a++month's price or a trend.++**Open Food Facts labels** — Open Food Facts' daily product export (ODbL), read by OpenVibe.Sources as a streamedbulk dataset (source `open-food-facts`), is matched by product name to the 42 foods on the list. A food's page showsthe most complete matching product's per-100 g figures, its Nutri-Score and a link to the product, in a separatennew file mode 100644@@@ -0,0 +1,14 @@++-- phase: expand++-- OpenVibe.Food's USDA ERS prices (server/food/usda-prices.js): the Economic Research Service's annual national-average++-- retail prices for fruits and vegetables, as OpenVibe.Sources collects them (source usda-ers-fruit-vegetable-prices,++-- category food). Food never calls ERS: it follows Sources' change feed from its own cursor in food_ingest_cursor and++-- keeps each row as ERS publishes it. US federal data (public domain in the US); nothing here belongs to a person.++CREATE TABLE food_usda_prices (++ identity text COLLATE "C" PRIMARY KEY, -- "Apples | Fresh": Sources' identity (ERS product and form)++ item_id text COLLATE "C" NOT NULL UNIQUE, -- the Sources item++ revision integer NOT NULL, -- the Sources item's revision this row holds++ value numeric(10, 4) NOT NULL CHECK (value > 0), -- dollars per the row's unit, as ERS states it++ unit text COLLATE "C" NOT NULL, -- lb (every row Food maps is priced per pound)++ vintage text COLLATE "C" NOT NULL, -- the data year ERS published (2023 today)++ retrieved_at text COLLATE "C" NOT NULL -- when Sources last read it, ISO 8601 UTC++);@@@ -98,6 +98,8 @@ async function createApp(opts = {}) {ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });++ // USDA ERS's annual fruit and vegetable prices beside BLS (server/food/usda-prices.js); opts.usdaPriceSource.++ ctx.usdaPrices = require('./food/usda-prices').createUsdaPrices({ config, s, fetchImpl, log, source: opts.usdaPriceSource || null });// Open Food Facts products as label data beside the list, from OpenVibe.Sources (server/food/off.js); opts.offSource.ctx.off = require('./food/off').createOff({ config, s, fetchImpl, log, source: opts.offSource || null });nnew file mode 100644@@@ -0,0 +1,173 @@++'use strict';++++/**++ * USDA prices: the Economic Research Service's annual national-average retail prices for fruits and vegetables, read++ * from OpenVibe.Sources (source usda-ers-fruit-vegetable-prices, category food), never from ERS. Sources reads ERS's++ * two published all-fruits and all-vegetables tables once a week under its robots and terms checks and keeps one item++ * per product and form (the identity is "product | form"); Food follows Sources' change feed from its own cursor++ * (openvibe-publishing/ingest, the same food_ingest_cursor table as the recalls and BLS) and keeps one row per++ * Sources identity in food_usda_prices.++ *++ * The food list's own prices stay what they are, a hand-kept 2025 band to plan by; a food's page shows the USDA figure++ * beside the BLS one, for ERS's per-pound price and worked out for the item's pack. ERS publishes annual averages, not++ * a monthly series, and says the data "should not be used for making inferences about price changes over time": the++ * page says the data year (2023) and NOTE repeats both, so the figure is never read as a trend or as this month's price.++ *++ * Pulls a minute after start and then every six hours. Off without OV_SOURCES_INTERNAL_URL and the OAuth client secret++ * (a test passes its own source).++ */++const ingest = require('openvibe-publishing/ingest');++++const SOURCE_KEY = 'usda-ers-fruit-vegetable-prices';++const CURSOR = 'sources.food.usda.after';++const EVERY_MS = 6 * 60 * 60 * 1000;++const FIRST_MS = 60 * 1000;++// ERS's current tables use 2023 retail scanner data; the files were last updated 2025-12-09. ERS updates them each++// year, subject to data availability: re-check both when the source's page changes (the seed's terms_note says so too).++const VINTAGE = '2023';++const URL = 'https://www.ers.usda.gov/data-products/fruit-and-vegetable-prices/';++// In plain words for a shopper: whose figure it is, that it is one year's national average, and that it is not a++// trend (ERS: "should not be used for making inferences about price changes over time"). ERS also asks that its++// findings not be attributed to Circana, whose scanner data it uses: the page never names Circana, which meets that.++const NOTE = 'A yearly national average from the USDA Economic Research Service (public domain): a snapshot of one year, not a price trend, so it is not compared with other years.';++++// Sources identity ("ERS product | form") → the food it prices, how many pounds the food's pack holds (null when the++// pack states no weight), and the plain words the page uses for the product. Every row mapped here is priced per pound.++const PRODUCTS = {++ 'Apples | Fresh': { slug: 'apples-3lb', pack: 3, label: 'apples' },++ 'Bananas | Fresh': { slug: 'bananas', pack: 1, label: 'bananas' },++ 'Oranges | Fresh': { slug: 'oranges-3lb', pack: 3, label: 'oranges' },++ 'Strawberries | Fresh': { slug: 'strawberries', pack: 1, label: 'strawberries' },++ 'Potatoes | Fresh': { slug: 'potatoes-5lb', pack: 5, label: 'potatoes' },++ 'Onions | Fresh': { slug: 'onion-3lb', pack: 3, label: 'onions' },++ 'Carrots, baby | Fresh': { slug: 'baby-carrots', pack: 1, label: 'baby carrots' },++ 'Avocados | Fresh': { slug: 'avocado', pack: null, label: 'avocados' },++ 'Grapes (raisins) | Dried': { slug: 'raisins', pack: 0.375, label: 'raisins' },++ 'Peaches, in syrup or water | Canned': { slug: 'canned-fruit', pack: 15 / 16, label: 'peaches in syrup or water' },++ 'Pinto beans | Dried': { slug: 'dried-beans', pack: 1, label: 'pinto beans' },++ 'Black beans | Canned': { slug: 'canned-beans', pack: 15 / 16, label: 'black beans' },++ 'Corn | Canned': { slug: 'canned-corn', pack: 15 / 16, label: 'corn' },++ 'Green beans | Canned': { slug: 'canned-green-beans', pack: 14.5 / 16, label: 'green beans' },++ 'Mixed vegetables, carrots, peas, corn, and green beans | Frozen': { slug: 'frozen-mixed-veg', pack: 12 / 16, label: 'mixed vegetables' },++ 'Tomatoes | Canned': { slug: 'canned-tomatoes', pack: 14.5 / 16, label: 'tomatoes' },++};++++const text = (v, max) => (v == null || v === '' ? null : String(v).replace(/\s+/g, ' ').trim().slice(0, max) || null);++++/** A Sources item → a food_usda_prices row, or null when it is not a row this list maps (or its unit is not a pound). */++function rowOf(item) {++ const f = (item && item.fields) || {};++ const identity = text(item && item.identity, 80);++ if (!identity || !Object.prototype.hasOwnProperty.call(PRODUCTS, identity)) return null;++ const value = Number(f.price);++ // Every mapped food is priced per pound; any other unit would make the pack arithmetic wrong, so hold it.++ if (!Number.isFinite(value) || value <= 0 || value > 1000 || text(f.unit, 20) !== 'per pound') return null;++ return {++ identity,++ item_id: String(item.id),++ revision: Number(item.revision) || 1,++ value: Math.round(value * 10000) / 10000,++ unit: 'lb',++ vintage: VINTAGE,++ retrieved_at: (item.provenance && item.provenance.retrieved_at) || new Date(0).toISOString(),++ };++}++++/** What a food's page shows for one row: ERS's figure and data year, and the price worked out for the food's pack. */++function shape(row) {++ const def = PRODUCTS[row.identity];++ const value = Number(row.value);++ return {++ slug: def.slug,++ identity: row.identity,++ item: def.label,++ form: row.identity.slice(row.identity.indexOf(' | ') + 3).toLowerCase(),++ unit: 'lb',++ value,++ pack_price: def.pack == null ? null : Math.round(value * def.pack * 100) / 100,++ year: row.vintage,++ url: URL,++ };++}++++function createUsdaPrices({ config, s, fetchImpl = globalThis.fetch, log = console, source = null }) {++ const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');++ const client = source || (url && config.oauth.clientSecret++ ? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'food' } }, fetchImpl })++ : null);++ const cursor = ingest.createChangeCursor(s.db, { prefix: 'food', now: s.now });++ const stats = { pulls: 0, applied: 0, removed: 0, last_at: null, last_error: null };++ let byFood = new Map(); // food slug → shape(row), refreshed after each pull++ let timer = null;++ let first = null;++ let running = false;++++ async function apply(item, t) {++ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'price') return 'hold';++ if (item.removed) {++ await t.query('DELETE FROM food_usda_prices WHERE item_id = $1', [String(item.id)]);++ return 'removed';++ }++ const row = rowOf(item);++ if (!row) return 'hold';++ const cols = Object.keys(row);++ // A replayed page (or an older revision arriving late) never takes a row back.++ await t.query(`INSERT INTO food_usda_prices (${cols.join(', ')}) VALUES (${cols.map((_, i) => `$${i + 1}`).join(', ')})++ ON CONFLICT (identity) DO UPDATE SET ${cols.filter((c) => c !== 'identity').map((c) => `${c} = EXCLUDED.${c}`).join(', ')}++ WHERE food_usda_prices.item_id <> EXCLUDED.item_id OR food_usda_prices.revision <= EXCLUDED.revision`, cols.map((c) => row[c]));++ return 'applied';++ }++++ async function load() {++ const rows = await s.db.many('SELECT identity, value, unit, vintage FROM food_usda_prices');++ byFood = new Map(rows.filter((r) => PRODUCTS[r.identity]).map((r) => [PRODUCTS[r.identity].slug, shape(r)]));++ return byFood.size;++ }++++ /** Follow Sources' change feed to its end. Never throws: the caller is a timer or a test. */++ async function pull() {++ if (!client || running) return { skipped: true };++ running = true;++ try {++ const out = await ingest.pullChanges({ db: s.db, cursor, source: client, name: CURSOR, apply });++ stats.pulls++;++ stats.applied += out.applied;++ stats.removed += out.removed;++ stats.last_at = s.iso();++ stats.last_error = null;++ await load();++ if (out.applied || out.removed) log.log(`[Prices] ${out.applied} USDA rows updated, ${out.removed} removed (Sources cursor ${out.after})`);++ return out;++ } catch (err) {++ stats.last_error = text(err && err.message, 300);++ log.warn(`[Prices] pull of USDA prices from OpenVibe.Sources failed: ${stats.last_error}`);++ return { error: stats.last_error };++ } finally { running = false; }++ }++++ return {++ enabled: Boolean(client),++ stats,++ pull,++ apply,++ load,++ /** The USDA figure for a food on the list, or null when ERS prices nothing like it (or nothing is read yet). */++ forFood(slug) { return byFood.get(slug) || null; },++ start() {++ load().catch(() => { /* the first pull loads it */ });++ if (!client || timer) return;++ first = setTimeout(() => { pull(); }, FIRST_MS);++ timer = setInterval(() => { pull(); }, EVERY_MS);++ if (first.unref) first.unref();++ if (timer.unref) timer.unref();++ },++ stop() {++ if (first) clearTimeout(first);++ if (timer) clearInterval(timer);++ first = null;++ timer = null;++ },++ };++}++++module.exports = { createUsdaPrices, rowOf, shape, PRODUCTS, SOURCE_KEY, NOTE, VINTAGE, URL };@@@ -30,6 +30,7 @@ const { markdown } = require('../render/markdown');const { send } = require('../render/layout');const foods = require('../food/foods');const { NOTE: OFFICIAL_NOTE } = require('../food/official-prices');++const { NOTE: USDA_NOTE } = require('../food/usda-prices');const { planMeals } = require('../food/planner');const { searchPlaces, safeUrl, ATTRIBUTION, NEAR_CITIES } = require('../food/places');const { matchPantry } = require('../food/pantry-match');@@@ -594,6 +595,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'r.get('/foods/:slug', (req, res) => {const f = foods.getFood(req.params.slug);const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;++ const usda = f && ctx.usdaPrices ? ctx.usdaPrices.forFood(f.slug) : null;const off = f && ctx.off ? ctx.off.forFood(f.slug) : null;if (!f) return page(req, res, {title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],@@@ -621,6 +623,11 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])${official ? html`<h2>US average price</h2><p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p><p class="muted small">${OFFICIAL_NOTE}</p>` : ''}++${usda ? html`<h2>USDA average price</h2>++<p>${usda.pack_price == null++ ? html`<b>${money(usda.value)}</b> per ${usda.unit} of ${usda.item} (${usda.form}) at the USDA ERS national average for ${usda.year} (<a href="${usda.url}" rel="noopener">USDA ERS Fruit and Vegetable Prices</a>).`++ : html`<b>${money(usda.pack_price)}</b> for this pack at the USDA ERS national average for ${usda.year}: ${money(usda.value)} per ${usda.unit} of ${usda.item} (${usda.form}) (<a href="${usda.url}" rel="noopener">USDA ERS Fruit and Vegetable Prices</a>).`}</p>++<p class="muted small">${USDA_NOTE}</p>` : ''}${off ? html`<h2>Open Food Facts label</h2><p><b>${off.name}</b>${off.brands ? html` · ${off.brands}` : ''}${off.nutriscore_grade ? html` · Nutri-Score ${off.nutriscore_grade}` : ''}</p>${table(['Per 100 g', 'As the label states'], [@@@ -47,14 +47,15 @@ async function start() {// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.ctx.library.start();ctx.recipeSearch.start();-- // Food recalls, BLS's average prices and Open Food Facts products follow OpenVibe.Sources' change feed-- // (server/food/recalls.js, server/food/official-prices.js, server/food/off.js).++ // Food recalls, BLS's average prices, USDA ERS's prices and Open Food Facts products follow OpenVibe.Sources'++ // change feed (server/food/recalls.js, server/food/official-prices.js, server/food/usda-prices.js, server/food/off.js).ctx.recalls.start();ctx.officialPrices.start();++ ctx.usdaPrices.start();ctx.off.start();// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).ctx.nearWarm.start();-- const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];++ const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.usdaPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];createLifecycle({ server, ctx, extra });return { server, ctx };}@@@ -50,7 +50,7 @@ async function boot(opts = {}) {const { createStore } = require('../../server/db');const testdb = await require('./db').testDb();const store = createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, offSource: opts.offSource });++ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, usdaPriceSource: opts.usdaPriceSource, offSource: opts.offSource });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;nnew file mode 100644@@@ -0,0 +1,117 @@++'use strict';++/**++ * USDA ERS prices (server/food/usda-prices.js): the Economic Research Service's annual fruit and vegetable prices from++ * OpenVibe.Sources.++ * - a usda-ers-fruit-vegetable-prices item for a mapped product and form becomes one row; another identity, unit,++ * kind or source is held;++ * - the pull follows the change feed from its own cursor (BLS and the recalls keep theirs), a removed item is++ * deleted, an older revision never wins;++ * - a food's page shows the figure beside the BLS one with its data year and the ERS note; a food ERS does not price++ * shows none.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const foods = require('../server/food/foods');++const { rowOf, shape, PRODUCTS, VINTAGE } = require('../server/food/usda-prices');++++const usdaItem = (identity, value, n, over = {}) => ({++ id: `itm_u${n}`, source_key: 'usda-ers-fruit-vegetable-prices', category: 'food', kind: 'price', identity,++ canonical_url: null, title: identity.split(' | ')[0], summary: null,++ fields: { name: identity.split(' | ')[0], form: identity.split(' | ')[1], price: String(value), unit: 'per pound' },++ revision: 1, provenance: { retrieved_at: '2026-11-05T06:00:00.000Z' }, removed: null, change_seq: n, ...over,++});++++const blsItem = (series, value, n) => ({++ id: 'itm_b1', source_key: 'bls-average-prices', category: 'food', kind: 'price', identity: series, canonical_url: null,++ title: series, summary: null, fields: { value: String(value), year: '2026', period: 'M09' }, revision: 1,++ provenance: { retrieved_at: '2026-11-05T06:00:00.000Z' }, removed: null, change_seq: n,++});++++function feed(items) {++ return {++ items,++ async listItems({ after = 0, limit = 100 }) {++ const rest = items.filter((x) => x.change_seq > after).sort((a, b) => a.change_seq - b.change_seq);++ const page = rest.slice(0, limit);++ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: rest.length > limit, sources: {} };++ },++ };++}++++(async () => {++ const usda = feed([++ usdaItem('Apples | Fresh', 1.86093095, 1),++ usdaItem('Bananas | Fresh', 0.609594742, 2),++ usdaItem('Strawberries | Fresh', 3.078734363, 3),++ usdaItem('Avocados | Fresh', 2.262402684, 4),++ usdaItem('Kale | Fresh', 3.57, 5), // a row Food does not map++ { ...usdaItem('Oranges | Fresh', 1.44, 6), source_key: 'bls-average-prices' }, // another source++ ]);++ const bls = feed([blsItem('APU0000711211', 0.633, 1)]);++ const t = await boot({ priceSource: bls, usdaPriceSource: usda });++ try {++ await check('every mapped identity prices a food on the list; a row is the item as ERS states it', async () => {++ for (const [identity, def] of Object.entries(PRODUCTS)) assert.ok(foods.getFood(def.slug), `${identity} → ${def.slug}`);++ const row = rowOf(usdaItem('Apples | Fresh', 1.86093095, 1));++ assert.deepStrictEqual([row.identity, row.value, row.unit, row.vintage], ['Apples | Fresh', 1.8609, 'lb', VINTAGE]);++ assert.strictEqual(rowOf(usdaItem('Kale | Fresh', 3.57, 1)), null, 'a row nobody maps');++ assert.strictEqual(rowOf(usdaItem('Apples | Fresh', 1.86, 1, { fields: { name: 'Apples', form: 'Fresh', price: '1.86', unit: 'per pint' } })), null, 'per pint would make the pack arithmetic wrong');++ assert.strictEqual(rowOf(usdaItem('Apples | Fresh', -1, 1)), null);++ const apples = shape({ identity: 'Apples | Fresh', value: '1.8609', unit: 'lb', vintage: '2023' });++ assert.deepStrictEqual([apples.slug, apples.item, apples.form, apples.pack_price, apples.year], ['apples-3lb', 'apples', 'fresh', 5.58, '2023'], 'a 3 lb bag');++ assert.strictEqual(shape({ identity: 'Strawberries | Fresh', value: '3.0787', unit: 'lb', vintage: '2023' }).pack_price, 3.08, 'a 1 lb pack');++ assert.strictEqual(shape({ identity: 'Grapes (raisins) | Dried', value: '4.3283', unit: 'lb', vintage: '2023' }).pack_price, 1.62, 'a 6 oz box is 0.375 lb');++ assert.strictEqual(shape({ identity: 'Avocados | Fresh', value: '2.2624', unit: 'lb', vintage: '2023' }).pack_price, null, 'a pack that states no weight gets no pack price');++ });++++ await check('the pull follows its own cursor; anything else is held; BLS keeps its own cursor', async () => {++ assert.strictEqual((await t.get('/foods/apples-3lb')).text.includes('USDA average price'), false, 'nothing before the first pull');++ const out = await t.ctx.usdaPrices.pull();++ assert.deepStrictEqual([out.applied, out.hold], [4, 2]);++ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.usda.after'")), 6);++ assert.strictEqual(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.prices.after'"), null, 'the BLS cursor is theirs');++ assert.deepStrictEqual([t.ctx.usdaPrices.forFood('apples-3lb').value, t.ctx.usdaPrices.forFood('apples-3lb').year], [1.8609, '2023']);++ assert.strictEqual(t.ctx.usdaPrices.forFood('white-rice'), null, 'a food ERS does not price');++ const blsOut = await t.ctx.officialPrices.pull();++ assert.strictEqual(blsOut.applied, 1);++ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.prices.after'")), 1);++ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.usda.after'")), 6);++ });++++ await check('a new value replaces the old; an older revision arriving late and a removed item behave', async () => {++ usda.items.push(usdaItem('Apples | Fresh', 2.0, 7, { id: 'itm_u1', revision: 3 })); // the same Sources item, a new value++ usda.items.push(usdaItem('Apples | Fresh', 9.99, 8, { id: 'itm_u1', revision: 2 })); // a replayed older revision++ usda.items.push({ ...usdaItem('Strawberries | Fresh', 3.0787, 9), id: 'itm_u3', removed: { at: '2026-11-05T07:00:00.000Z', reason: 'replaced' }, revision: 2 });++ const out = await t.ctx.usdaPrices.pull();++ assert.strictEqual(out.removed, 1);++ assert.strictEqual(t.ctx.usdaPrices.forFood('apples-3lb').value, 2);++ assert.strictEqual(t.ctx.usdaPrices.forFood('strawberries'), null);++ });++++ await check('a food\'s page shows the USDA figure with its data year and the ERS note beside BLS', async () => {++ const apples = await t.get('/foods/apples-3lb');++ assert.strictEqual(apples.status, 200);++ assert.ok(apples.text.includes('USDA average price'));++ assert.ok(apples.text.includes('$6.00</b> for this pack at the USDA ERS national average for 2023: $2.00 per lb of apples (fresh)'), apples.text.slice(apples.text.indexOf('USDA average'), apples.text.indexOf('USDA average') + 400));++ assert.ok(apples.text.includes('https://www.ers.usda.gov/data-products/fruit-and-vegetable-prices/'));++ assert.ok(apples.text.includes('a snapshot of one year, not a price trend'), 'ERS: no inference about change over time');++ assert.ok(!/Circana/.test(apples.text), 'ERS: the findings are not attributed to Circana (the page never names it)');++ assert.ok(!apples.text.includes('US average price'), 'BLS prices no apples');++ const bananas = await t.get('/foods/bananas');++ assert.ok(bananas.text.includes('US average price'), 'BLS on the same page');++ assert.ok(bananas.text.includes('September 2026'), 'BLS month');++ assert.ok(bananas.text.includes('USDA average price'));++ assert.ok(bananas.text.includes('for 2023'), 'USDA data year');++ assert.ok(bananas.text.includes('per lb of bananas (fresh)'));++ const avocado = await t.get('/foods/avocado');++ assert.ok(avocado.text.includes('$2.26</b> per lb of avocados (fresh)'), 'no pack weight, so no pack price');++ assert.ok(!avocado.text.includes('for this pack at the USDA'), 'no invented pack weight');++ const tuna = await t.get('/foods/canned-tuna');++ assert.ok(!tuna.text.includes('USDA average price'));++ assert.ok(!tuna.text.includes('US average price'));++ });++ } finally {++ await t.close();++ }++ done();++})();
@@ -167,6 +167,15 @@ published average-price file once a day (source `bls-average-prices`) and Food f
@@ -167,6 +167,15 @@ published average-price file once a day (source `bls-average-prices`) and Food f
`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list's
`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list's
own band stays as it is.
own band stays as it is.
+**USDA prices** — the U.S. Department of Agriculture's Economic Research Service publishes annual national-average
+retail prices for 155 fresh and processed fruits and vegetables (Fruit and Vegetable Prices, 2023 retail scanner data,
+public domain); 16 of Food's items are among them, and an item's page shows ERS's figure for the food's pack beside
+BLS's where both price it ([server/food/usda-prices.js](server/food/usda-prices.js)). Food never calls ERS:
+OpenVibe.Sources reads the two published CSV tables once a week (source `usda-ers-fruit-vegetable-prices`) and Food
+follows its change feed into `food_usda_prices`. The page gives the data year (2023) and repeats ERS's warning that the
+data should not be used for making inferences about price changes over time; the annual figure is never shown as a
+month's price or a trend.
+
**Open Food Facts labels** — Open Food Facts' daily product export (ODbL), read by OpenVibe.Sources as a streamed
**Open Food Facts labels** — Open Food Facts' daily product export (ODbL), read by OpenVibe.Sources as a streamed
bulk dataset (source `open-food-facts`), is matched by product name to the 42 foods on the list. A food's page shows
bulk dataset (source `open-food-facts`), is matched by product name to the 42 foods on the list. A food's page shows
the most complete matching product's per-100 g figures, its Nutri-Score and a link to the product, in a separate
the most complete matching product's per-100 g figures, its Nutri-Score and a link to the product, in a separate
new file mode 100644
new file mode 100644
@@ -0,0 +1,14 @@
@@ -0,0 +1,14 @@
+-- phase: expand
+-- OpenVibe.Food's USDA ERS prices (server/food/usda-prices.js): the Economic Research Service's annual national-average
+-- retail prices for fruits and vegetables, as OpenVibe.Sources collects them (source usda-ers-fruit-vegetable-prices,
+-- category food). Food never calls ERS: it follows Sources' change feed from its own cursor in food_ingest_cursor and
+-- keeps each row as ERS publishes it. US federal data (public domain in the US); nothing here belongs to a person.
+CREATE TABLE food_usda_prices (
+ identity text COLLATE "C" PRIMARY KEY, -- "Apples | Fresh": Sources' identity (ERS product and form)
+ item_id text COLLATE "C" NOT NULL UNIQUE, -- the Sources item
+ revision integer NOT NULL, -- the Sources item's revision this row holds
+ value numeric(10, 4) NOT NULL CHECK (value > 0), -- dollars per the row's unit, as ERS states it
+ unit text COLLATE "C" NOT NULL, -- lb (every row Food maps is priced per pound)
+ vintage text COLLATE "C" NOT NULL, -- the data year ERS published (2023 today)
+ retrieved_at text COLLATE "C" NOT NULL -- when Sources last read it, ISO 8601 UTC
+);
@@ -98,6 +98,8 @@ async function createApp(opts = {}) {
@@ -98,6 +98,8 @@ async function createApp(opts = {}) {
ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });
ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });
// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.
// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.
ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });
ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });
+ // USDA ERS's annual fruit and vegetable prices beside BLS (server/food/usda-prices.js); opts.usdaPriceSource.
+ ctx.usdaPrices = require('./food/usda-prices').createUsdaPrices({ config, s, fetchImpl, log, source: opts.usdaPriceSource || null });
// Open Food Facts products as label data beside the list, from OpenVibe.Sources (server/food/off.js); opts.offSource.
// Open Food Facts products as label data beside the list, from OpenVibe.Sources (server/food/off.js); opts.offSource.
ctx.off = require('./food/off').createOff({ config, s, fetchImpl, log, source: opts.offSource || null });
ctx.off = require('./food/off').createOff({ config, s, fetchImpl, log, source: opts.offSource || null });
new file mode 100644
new file mode 100644
@@ -0,0 +1,173 @@
@@ -0,0 +1,173 @@
+'use strict';
+
+/**
+ * USDA prices: the Economic Research Service's annual national-average retail prices for fruits and vegetables, read
+ * from OpenVibe.Sources (source usda-ers-fruit-vegetable-prices, category food), never from ERS. Sources reads ERS's
+ * two published all-fruits and all-vegetables tables once a week under its robots and terms checks and keeps one item
+ * per product and form (the identity is "product | form"); Food follows Sources' change feed from its own cursor
+ * (openvibe-publishing/ingest, the same food_ingest_cursor table as the recalls and BLS) and keeps one row per
+ * Sources identity in food_usda_prices.
+ *
+ * The food list's own prices stay what they are, a hand-kept 2025 band to plan by; a food's page shows the USDA figure
+ * beside the BLS one, for ERS's per-pound price and worked out for the item's pack. ERS publishes annual averages, not
+ * a monthly series, and says the data "should not be used for making inferences about price changes over time": the
+ * page says the data year (2023) and NOTE repeats both, so the figure is never read as a trend or as this month's price.
+ *
+ * Pulls a minute after start and then every six hours. Off without OV_SOURCES_INTERNAL_URL and the OAuth client secret
+ * (a test passes its own source).
+ */
+const ingest = require('openvibe-publishing/ingest');
+
+const SOURCE_KEY = 'usda-ers-fruit-vegetable-prices';
+const CURSOR = 'sources.food.usda.after';
+const EVERY_MS = 6 * 60 * 60 * 1000;
+const FIRST_MS = 60 * 1000;
+// ERS's current tables use 2023 retail scanner data; the files were last updated 2025-12-09. ERS updates them each
+// year, subject to data availability: re-check both when the source's page changes (the seed's terms_note says so too).
+const VINTAGE = '2023';
+const URL = 'https://www.ers.usda.gov/data-products/fruit-and-vegetable-prices/';
+// In plain words for a shopper: whose figure it is, that it is one year's national average, and that it is not a
+// trend (ERS: "should not be used for making inferences about price changes over time"). ERS also asks that its
+// findings not be attributed to Circana, whose scanner data it uses: the page never names Circana, which meets that.
+const NOTE = 'A yearly national average from the USDA Economic Research Service (public domain): a snapshot of one year, not a price trend, so it is not compared with other years.';
+
+// Sources identity ("ERS product | form") → the food it prices, how many pounds the food's pack holds (null when the
+// pack states no weight), and the plain words the page uses for the product. Every row mapped here is priced per pound.
+const PRODUCTS = {
+ 'Apples | Fresh': { slug: 'apples-3lb', pack: 3, label: 'apples' },
+ 'Bananas | Fresh': { slug: 'bananas', pack: 1, label: 'bananas' },
+ 'Oranges | Fresh': { slug: 'oranges-3lb', pack: 3, label: 'oranges' },
+ 'Strawberries | Fresh': { slug: 'strawberries', pack: 1, label: 'strawberries' },
+ 'Potatoes | Fresh': { slug: 'potatoes-5lb', pack: 5, label: 'potatoes' },
+ 'Onions | Fresh': { slug: 'onion-3lb', pack: 3, label: 'onions' },
+ 'Carrots, baby | Fresh': { slug: 'baby-carrots', pack: 1, label: 'baby carrots' },
+ 'Avocados | Fresh': { slug: 'avocado', pack: null, label: 'avocados' },
+ 'Grapes (raisins) | Dried': { slug: 'raisins', pack: 0.375, label: 'raisins' },
+ 'Peaches, in syrup or water | Canned': { slug: 'canned-fruit', pack: 15 / 16, label: 'peaches in syrup or water' },
+ 'Pinto beans | Dried': { slug: 'dried-beans', pack: 1, label: 'pinto beans' },
+ 'Black beans | Canned': { slug: 'canned-beans', pack: 15 / 16, label: 'black beans' },
+ 'Corn | Canned': { slug: 'canned-corn', pack: 15 / 16, label: 'corn' },
+ 'Green beans | Canned': { slug: 'canned-green-beans', pack: 14.5 / 16, label: 'green beans' },
+ 'Mixed vegetables, carrots, peas, corn, and green beans | Frozen': { slug: 'frozen-mixed-veg', pack: 12 / 16, label: 'mixed vegetables' },
+ 'Tomatoes | Canned': { slug: 'canned-tomatoes', pack: 14.5 / 16, label: 'tomatoes' },
+};
+
+const text = (v, max) => (v == null || v === '' ? null : String(v).replace(/\s+/g, ' ').trim().slice(0, max) || null);
+
+/** A Sources item → a food_usda_prices row, or null when it is not a row this list maps (or its unit is not a pound). */
+function rowOf(item) {
+ const f = (item && item.fields) || {};
+ const identity = text(item && item.identity, 80);
+ if (!identity || !Object.prototype.hasOwnProperty.call(PRODUCTS, identity)) return null;
+ const value = Number(f.price);
+ // Every mapped food is priced per pound; any other unit would make the pack arithmetic wrong, so hold it.
+ if (!Number.isFinite(value) || value <= 0 || value > 1000 || text(f.unit, 20) !== 'per pound') return null;
+ return {
+ identity,
+ item_id: String(item.id),
+ revision: Number(item.revision) || 1,
+ value: Math.round(value * 10000) / 10000,
+ unit: 'lb',
+ vintage: VINTAGE,
+ retrieved_at: (item.provenance && item.provenance.retrieved_at) || new Date(0).toISOString(),
+ };
+}
+
+/** What a food's page shows for one row: ERS's figure and data year, and the price worked out for the food's pack. */
+function shape(row) {
+ const def = PRODUCTS[row.identity];
+ const value = Number(row.value);
+ return {
+ slug: def.slug,
+ identity: row.identity,
+ item: def.label,
+ form: row.identity.slice(row.identity.indexOf(' | ') + 3).toLowerCase(),
+ unit: 'lb',
+ value,
+ pack_price: def.pack == null ? null : Math.round(value * def.pack * 100) / 100,
+ year: row.vintage,
+ url: URL,
+ };
+}
+
+function createUsdaPrices({ config, s, fetchImpl = globalThis.fetch, log = console, source = null }) {
+ const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');
+ const client = source || (url && config.oauth.clientSecret
+ ? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'food' } }, fetchImpl })
+ : null);
+ const cursor = ingest.createChangeCursor(s.db, { prefix: 'food', now: s.now });
+ const stats = { pulls: 0, applied: 0, removed: 0, last_at: null, last_error: null };
+ let byFood = new Map(); // food slug → shape(row), refreshed after each pull
+ let timer = null;
+ let first = null;
+ let running = false;
+
+ async function apply(item, t) {
+ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'price') return 'hold';
+ if (item.removed) {
+ await t.query('DELETE FROM food_usda_prices WHERE item_id = $1', [String(item.id)]);
+ return 'removed';
+ }
+ const row = rowOf(item);
+ if (!row) return 'hold';
+ const cols = Object.keys(row);
+ // A replayed page (or an older revision arriving late) never takes a row back.
+ await t.query(`INSERT INTO food_usda_prices (${cols.join(', ')}) VALUES (${cols.map((_, i) => `$${i + 1}`).join(', ')})
+ ON CONFLICT (identity) DO UPDATE SET ${cols.filter((c) => c !== 'identity').map((c) => `${c} = EXCLUDED.${c}`).join(', ')}
+ WHERE food_usda_prices.item_id <> EXCLUDED.item_id OR food_usda_prices.revision <= EXCLUDED.revision`, cols.map((c) => row[c]));
+ return 'applied';
+ }
+
+ async function load() {
+ const rows = await s.db.many('SELECT identity, value, unit, vintage FROM food_usda_prices');
+ byFood = new Map(rows.filter((r) => PRODUCTS[r.identity]).map((r) => [PRODUCTS[r.identity].slug, shape(r)]));
+ return byFood.size;
+ }
+
+ /** Follow Sources' change feed to its end. Never throws: the caller is a timer or a test. */
+ async function pull() {
+ if (!client || running) return { skipped: true };
+ running = true;
+ try {
+ const out = await ingest.pullChanges({ db: s.db, cursor, source: client, name: CURSOR, apply });
+ stats.pulls++;
+ stats.applied += out.applied;
+ stats.removed += out.removed;
+ stats.last_at = s.iso();
+ stats.last_error = null;
+ await load();
+ if (out.applied || out.removed) log.log(`[Prices] ${out.applied} USDA rows updated, ${out.removed} removed (Sources cursor ${out.after})`);
+ return out;
+ } catch (err) {
+ stats.last_error = text(err && err.message, 300);
+ log.warn(`[Prices] pull of USDA prices from OpenVibe.Sources failed: ${stats.last_error}`);
+ return { error: stats.last_error };
+ } finally { running = false; }
+ }
+
+ return {
+ enabled: Boolean(client),
+ stats,
+ pull,
+ apply,
+ load,
+ /** The USDA figure for a food on the list, or null when ERS prices nothing like it (or nothing is read yet). */
+ forFood(slug) { return byFood.get(slug) || null; },
+ start() {
+ load().catch(() => { /* the first pull loads it */ });
+ if (!client || timer) return;
+ first = setTimeout(() => { pull(); }, FIRST_MS);
+ timer = setInterval(() => { pull(); }, EVERY_MS);
+ if (first.unref) first.unref();
+ if (timer.unref) timer.unref();
+ },
+ stop() {
+ if (first) clearTimeout(first);
+ if (timer) clearInterval(timer);
+ first = null;
+ timer = null;
+ },
+ };
+}
+
+module.exports = { createUsdaPrices, rowOf, shape, PRODUCTS, SOURCE_KEY, NOTE, VINTAGE, URL };
@@ -30,6 +30,7 @@ const { markdown } = require('../render/markdown');
@@ -30,6 +30,7 @@ const { markdown } = require('../render/markdown');
const { send } = require('../render/layout');
const { send } = require('../render/layout');
const foods = require('../food/foods');
const foods = require('../food/foods');
const { NOTE: OFFICIAL_NOTE } = require('../food/official-prices');
const { NOTE: OFFICIAL_NOTE } = require('../food/official-prices');
+const { NOTE: USDA_NOTE } = require('../food/usda-prices');
const { planMeals } = require('../food/planner');
const { planMeals } = require('../food/planner');
const { searchPlaces, safeUrl, ATTRIBUTION, NEAR_CITIES } = require('../food/places');
const { searchPlaces, safeUrl, ATTRIBUTION, NEAR_CITIES } = require('../food/places');
const { matchPantry } = require('../food/pantry-match');
const { matchPantry } = require('../food/pantry-match');
@@ -594,6 +595,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'
@@ -594,6 +595,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'
r.get('/foods/:slug', (req, res) => {
r.get('/foods/:slug', (req, res) => {
const f = foods.getFood(req.params.slug);
const f = foods.getFood(req.params.slug);
const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;
const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;
+ const usda = f && ctx.usdaPrices ? ctx.usdaPrices.forFood(f.slug) : null;
const off = f && ctx.off ? ctx.off.forFood(f.slug) : null;
const off = f && ctx.off ? ctx.off.forFood(f.slug) : null;
if (!f) return page(req, res, {
if (!f) return page(req, res, {
title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],
title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],
@@ -621,6 +623,11 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])
@@ -621,6 +623,11 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])
${official ? html`<h2>US average price</h2>
${official ? html`<h2>US average price</h2>
<p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p>
<p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p>
<p class="muted small">${OFFICIAL_NOTE}</p>` : ''}
<p class="muted small">${OFFICIAL_NOTE}</p>` : ''}
+${usda ? html`<h2>USDA average price</h2>
+<p>${usda.pack_price == null
+ ? html`<b>${money(usda.value)}</b> per ${usda.unit} of ${usda.item} (${usda.form}) at the USDA ERS national average for ${usda.year} (<a href="${usda.url}" rel="noopener">USDA ERS Fruit and Vegetable Prices</a>).`
+ : html`<b>${money(usda.pack_price)}</b> for this pack at the USDA ERS national average for ${usda.year}: ${money(usda.value)} per ${usda.unit} of ${usda.item} (${usda.form}) (<a href="${usda.url}" rel="noopener">USDA ERS Fruit and Vegetable Prices</a>).`}</p>
+<p class="muted small">${USDA_NOTE}</p>` : ''}
${off ? html`<h2>Open Food Facts label</h2>
${off ? html`<h2>Open Food Facts label</h2>
<p><b>${off.name}</b>${off.brands ? html` · ${off.brands}` : ''}${off.nutriscore_grade ? html` · Nutri-Score ${off.nutriscore_grade}` : ''}</p>
<p><b>${off.name}</b>${off.brands ? html` · ${off.brands}` : ''}${off.nutriscore_grade ? html` · Nutri-Score ${off.nutriscore_grade}` : ''}</p>
${table(['Per 100 g', 'As the label states'], [
${table(['Per 100 g', 'As the label states'], [
@@ -47,14 +47,15 @@ async function start() {
@@ -47,14 +47,15 @@ async function start() {
// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.
// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.
ctx.library.start();
ctx.library.start();
ctx.recipeSearch.start();
ctx.recipeSearch.start();
- // Food recalls, BLS's average prices and Open Food Facts products follow OpenVibe.Sources' change feed
- // (server/food/recalls.js, server/food/official-prices.js, server/food/off.js).
+ // Food recalls, BLS's average prices, USDA ERS's prices and Open Food Facts products follow OpenVibe.Sources'
+ // change feed (server/food/recalls.js, server/food/official-prices.js, server/food/usda-prices.js, server/food/off.js).
ctx.recalls.start();
ctx.recalls.start();
ctx.officialPrices.start();
ctx.officialPrices.start();
+ ctx.usdaPrices.start();
ctx.off.start();
ctx.off.start();
// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).
// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).
ctx.nearWarm.start();
ctx.nearWarm.start();
- const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];
+ const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.usdaPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];
createLifecycle({ server, ctx, extra });
createLifecycle({ server, ctx, extra });
return { server, ctx };
return { server, ctx };
}
}
@@ -50,7 +50,7 @@ async function boot(opts = {}) {
@@ -50,7 +50,7 @@ async function boot(opts = {}) {
const { createStore } = require('../../server/db');
const { createStore } = require('../../server/db');
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
const store = createStore(testdb.db, { now: opts.now });
const store = createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, offSource: opts.offSource });
+ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, usdaPriceSource: opts.usdaPriceSource, offSource: opts.offSource });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
new file mode 100644
new file mode 100644
@@ -0,0 +1,117 @@
@@ -0,0 +1,117 @@
+'use strict';
+/**
+ * USDA ERS prices (server/food/usda-prices.js): the Economic Research Service's annual fruit and vegetable prices from
+ * OpenVibe.Sources.
+ * - a usda-ers-fruit-vegetable-prices item for a mapped product and form becomes one row; another identity, unit,
+ * kind or source is held;
+ * - the pull follows the change feed from its own cursor (BLS and the recalls keep theirs), a removed item is
+ * deleted, an older revision never wins;
+ * - a food's page shows the figure beside the BLS one with its data year and the ERS note; a food ERS does not price
+ * shows none.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const foods = require('../server/food/foods');
+const { rowOf, shape, PRODUCTS, VINTAGE } = require('../server/food/usda-prices');
+
+const usdaItem = (identity, value, n, over = {}) => ({
+ id: `itm_u${n}`, source_key: 'usda-ers-fruit-vegetable-prices', category: 'food', kind: 'price', identity,
+ canonical_url: null, title: identity.split(' | ')[0], summary: null,
+ fields: { name: identity.split(' | ')[0], form: identity.split(' | ')[1], price: String(value), unit: 'per pound' },
+ revision: 1, provenance: { retrieved_at: '2026-11-05T06:00:00.000Z' }, removed: null, change_seq: n, ...over,
+});
+
+const blsItem = (series, value, n) => ({
+ id: 'itm_b1', source_key: 'bls-average-prices', category: 'food', kind: 'price', identity: series, canonical_url: null,
+ title: series, summary: null, fields: { value: String(value), year: '2026', period: 'M09' }, revision: 1,
+ provenance: { retrieved_at: '2026-11-05T06:00:00.000Z' }, removed: null, change_seq: n,
+});
+
+function feed(items) {
+ return {
+ items,
+ async listItems({ after = 0, limit = 100 }) {
+ const rest = items.filter((x) => x.change_seq > after).sort((a, b) => a.change_seq - b.change_seq);
+ const page = rest.slice(0, limit);
+ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: rest.length > limit, sources: {} };
+ },
+ };
+}
+
+(async () => {
+ const usda = feed([
+ usdaItem('Apples | Fresh', 1.86093095, 1),
+ usdaItem('Bananas | Fresh', 0.609594742, 2),
+ usdaItem('Strawberries | Fresh', 3.078734363, 3),
+ usdaItem('Avocados | Fresh', 2.262402684, 4),
+ usdaItem('Kale | Fresh', 3.57, 5), // a row Food does not map
+ { ...usdaItem('Oranges | Fresh', 1.44, 6), source_key: 'bls-average-prices' }, // another source
+ ]);
+ const bls = feed([blsItem('APU0000711211', 0.633, 1)]);
+ const t = await boot({ priceSource: bls, usdaPriceSource: usda });
+ try {
+ await check('every mapped identity prices a food on the list; a row is the item as ERS states it', async () => {
+ for (const [identity, def] of Object.entries(PRODUCTS)) assert.ok(foods.getFood(def.slug), `${identity} → ${def.slug}`);
+ const row = rowOf(usdaItem('Apples | Fresh', 1.86093095, 1));
+ assert.deepStrictEqual([row.identity, row.value, row.unit, row.vintage], ['Apples | Fresh', 1.8609, 'lb', VINTAGE]);
+ assert.strictEqual(rowOf(usdaItem('Kale | Fresh', 3.57, 1)), null, 'a row nobody maps');
+ assert.strictEqual(rowOf(usdaItem('Apples | Fresh', 1.86, 1, { fields: { name: 'Apples', form: 'Fresh', price: '1.86', unit: 'per pint' } })), null, 'per pint would make the pack arithmetic wrong');
+ assert.strictEqual(rowOf(usdaItem('Apples | Fresh', -1, 1)), null);
+ const apples = shape({ identity: 'Apples | Fresh', value: '1.8609', unit: 'lb', vintage: '2023' });
+ assert.deepStrictEqual([apples.slug, apples.item, apples.form, apples.pack_price, apples.year], ['apples-3lb', 'apples', 'fresh', 5.58, '2023'], 'a 3 lb bag');
+ assert.strictEqual(shape({ identity: 'Strawberries | Fresh', value: '3.0787', unit: 'lb', vintage: '2023' }).pack_price, 3.08, 'a 1 lb pack');
+ assert.strictEqual(shape({ identity: 'Grapes (raisins) | Dried', value: '4.3283', unit: 'lb', vintage: '2023' }).pack_price, 1.62, 'a 6 oz box is 0.375 lb');
+ assert.strictEqual(shape({ identity: 'Avocados | Fresh', value: '2.2624', unit: 'lb', vintage: '2023' }).pack_price, null, 'a pack that states no weight gets no pack price');
+ });
+
+ await check('the pull follows its own cursor; anything else is held; BLS keeps its own cursor', async () => {
+ assert.strictEqual((await t.get('/foods/apples-3lb')).text.includes('USDA average price'), false, 'nothing before the first pull');
+ const out = await t.ctx.usdaPrices.pull();
+ assert.deepStrictEqual([out.applied, out.hold], [4, 2]);
+ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.usda.after'")), 6);
+ assert.strictEqual(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.prices.after'"), null, 'the BLS cursor is theirs');
+ assert.deepStrictEqual([t.ctx.usdaPrices.forFood('apples-3lb').value, t.ctx.usdaPrices.forFood('apples-3lb').year], [1.8609, '2023']);
+ assert.strictEqual(t.ctx.usdaPrices.forFood('white-rice'), null, 'a food ERS does not price');
+ const blsOut = await t.ctx.officialPrices.pull();
+ assert.strictEqual(blsOut.applied, 1);
+ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.prices.after'")), 1);
+ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.usda.after'")), 6);
+ });
+
+ await check('a new value replaces the old; an older revision arriving late and a removed item behave', async () => {
+ usda.items.push(usdaItem('Apples | Fresh', 2.0, 7, { id: 'itm_u1', revision: 3 })); // the same Sources item, a new value
+ usda.items.push(usdaItem('Apples | Fresh', 9.99, 8, { id: 'itm_u1', revision: 2 })); // a replayed older revision
+ usda.items.push({ ...usdaItem('Strawberries | Fresh', 3.0787, 9), id: 'itm_u3', removed: { at: '2026-11-05T07:00:00.000Z', reason: 'replaced' }, revision: 2 });
+ const out = await t.ctx.usdaPrices.pull();
+ assert.strictEqual(out.removed, 1);
+ assert.strictEqual(t.ctx.usdaPrices.forFood('apples-3lb').value, 2);
+ assert.strictEqual(t.ctx.usdaPrices.forFood('strawberries'), null);
+ });
+
+ await check('a food\'s page shows the USDA figure with its data year and the ERS note beside BLS', async () => {
+ const apples = await t.get('/foods/apples-3lb');
+ assert.strictEqual(apples.status, 200);
+ assert.ok(apples.text.includes('USDA average price'));
+ assert.ok(apples.text.includes('$6.00</b> for this pack at the USDA ERS national average for 2023: $2.00 per lb of apples (fresh)'), apples.text.slice(apples.text.indexOf('USDA average'), apples.text.indexOf('USDA average') + 400));
+ assert.ok(apples.text.includes('https://www.ers.usda.gov/data-products/fruit-and-vegetable-prices/'));
+ assert.ok(apples.text.includes('a snapshot of one year, not a price trend'), 'ERS: no inference about change over time');
+ assert.ok(!/Circana/.test(apples.text), 'ERS: the findings are not attributed to Circana (the page never names it)');
+ assert.ok(!apples.text.includes('US average price'), 'BLS prices no apples');
+ const bananas = await t.get('/foods/bananas');
+ assert.ok(bananas.text.includes('US average price'), 'BLS on the same page');
+ assert.ok(bananas.text.includes('September 2026'), 'BLS month');
+ assert.ok(bananas.text.includes('USDA average price'));
+ assert.ok(bananas.text.includes('for 2023'), 'USDA data year');
+ assert.ok(bananas.text.includes('per lb of bananas (fresh)'));
+ const avocado = await t.get('/foods/avocado');
+ assert.ok(avocado.text.includes('$2.26</b> per lb of avocados (fresh)'), 'no pack weight, so no pack price');
+ assert.ok(!avocado.text.includes('for this pack at the USDA'), 'no invented pack weight');
+ const tuna = await t.get('/foods/canned-tuna');
+ assert.ok(!tuna.text.includes('USDA average price'));
+ assert.ok(!tuna.text.includes('US average price'));
+ });
+ } finally {
+ await t.close();
+ }
+ done();
+})();