openvibers/openvibe.food Commits

A food's page shows BLS's US city average price for the newest month and for its pack, for the 13 items BLS prices (#78)

OP OpenVibers committed 21 days ago · 46641e6147 · 8 files, +286 −3

Read from OpenVibe.Sources (source bls-average-prices, category food) on its own change-feed cursor into
food_official_prices (expand 0010), never from BLS; BLS is cited with its note. The list's own 2025 band
stays as it is. Idle until Sources serves the source.

Changes

ddiff --git a/README.md b/README.md
iindex 47a60e2..d5c85a8 100644
---- a/README.md
++++ b/README.md
@@@ -154,6 +154,14 @@ contributors, [ODbL 1.0](https://opendatacommons.org/licenses/odbl/1-0/). Read f
prices for five budget chains and the USDA MyPlate groups. They are a band to plan by, **never a live price**. The
Walmart page scraping that source also did is deliberately not ported: this service never scrapes a retail site.
++**Official prices** — for the 13 items the U.S. Bureau of Labor Statistics prices (eggs, boneless chicken breast,
++ground beef, white bread, rice, spaghetti, dried beans, bananas, oranges, strawberries, potatoes, whole milk, cheddar),
++an item's page also shows BLS's US city average for the newest month and that figure worked out for the item's pack
++([server/food/official-prices.js](server/food/official-prices.js)). Food never calls BLS: OpenVibe.Sources reads BLS's
++published average-price file once a day (source `bls-average-prices`) and Food follows its change feed into
++`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list's
++own band stays as it is.
++
**Nutrition** — the same source's typical values per serving. "Per 100 g" is arithmetic on the pack size stated in an
item's own name divided by its servings (for example a 15 oz can with 3.5 servings is 121 g a serving). Where a name
states no weight — a dozen eggs, a 12-pack of ramen — no per-100 g figure is given rather than a density guessed.
ddiff --git a/migrations/0010_official_prices.sql b/migrations/0010_official_prices.sql
nnew file mode 100644
iindex 0000000..d9911dd
---- /dev/null
++++ b/migrations/0010_official_prices.sql
@@@ -0,0 +1,15 @@
++-- phase: expand
++-- OpenVibe.Food's official prices (server/food/official-prices.js): the Bureau of Labor Statistics' monthly US city
++-- average retail prices for the items on the food list that BLS prices, as OpenVibe.Sources collects them (source
++-- bls-average-prices, category food). Food never calls BLS: it follows Sources' change feed from its own cursor in
++-- food_ingest_cursor and keeps each series' newest month as BLS states it. US federal data (public domain); nothing
++-- here belongs to a person.
++CREATE TABLE food_official_prices (
++ series_id text COLLATE "C" PRIMARY KEY, -- BLS's series (APU0000708111), Sources' identity
++ item_id text COLLATE "C" NOT NULL UNIQUE, -- the Sources item
++ revision integer NOT NULL, -- the Sources item's revision this row holds
++ value numeric(10, 3) NOT NULL CHECK (value > 0), -- dollars per the series' unit, as BLS states it
++ year integer NOT NULL,
++ period text COLLATE "C" NOT NULL, -- M01 … M12
++ retrieved_at text COLLATE "C" NOT NULL -- when Sources last read it, ISO 8601 UTC
++);
ddiff --git a/server/app.js b/server/app.js
iindex 9351324..0c8a303 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -92,6 +92,8 @@ async function createApp(opts = {}) {
ctx.recipeSearch = require('./food/search-index').createRecipeSearchIndex({ config, library: ctx.library, log });
// Food recalls from OpenVibe.Sources (server/food/recalls.js); a test passes a stand-in source through opts.recallSource.
ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });
++ // BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.
++ ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });
const app = express();
app.disable('x-powered-by');
ddiff --git a/server/food/official-prices.js b/server/food/official-prices.js
nnew file mode 100644
iindex 0000000..3973db9
---- /dev/null
++++ b/server/food/official-prices.js
@@@ -0,0 +1,161 @@
++'use strict';
++
++/**
++ * Official prices: the Bureau of Labor Statistics' monthly US city average retail prices for the items on the food list
++ * that BLS prices, read from OpenVibe.Sources (source bls-average-prices, category food), never from BLS. Sources reads
++ * BLS's published average-price file once a day under its robots and terms checks and keeps each series' newest month
++ * as an item; Food follows Sources' change feed from its own cursor (openvibe-publishing/ingest, the same
++ * food_ingest_cursor table as the recalls) and keeps one row per series in food_official_prices.
++ *
++ * The food list's own prices stay what they are, a hand-kept 2025 band to plan by; a food's page shows the BLS figure
++ * beside it, for the series' unit and worked out for the item's pack. BLS asks to be cited and says it cannot vouch for
++ * the data once they leave BLS.gov: NOTE says both wherever a figure shows.
++ *
++ * Pulls 45 seconds after start and then hourly. Off without OV_SOURCES_INTERNAL_URL and the OAuth client secret (a test
++ * passes its own source).
++ */
++const ingest = require('openvibe-publishing/ingest');
++
++const SOURCE_KEY = 'bls-average-prices';
++const CURSOR = 'sources.food.prices.after';
++const EVERY_MS = 60 * 60 * 1000;
++const FIRST_MS = 45 * 1000;
++const NOTE = 'US city average retail prices from the U.S. Bureau of Labor Statistics (Average Price Data, public domain). BLS.gov cannot vouch for the data or analyses derived from these data after the data have been retrieved from BLS.gov.';
++const MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
++
++// BLS series (US city average) → the food it prices, the series' unit, and how many units the food's pack holds.
++const SERIES = {
++ APU0000708111: { slug: 'eggs-dozen', unit: 'dozen', item: 'Eggs, grade A, large', perPack: 1 },
++ APU0000FF1101: { slug: 'chicken-breast', unit: 'lb', item: 'Chicken breast, boneless', perPack: 1 },
++ APU0000703112: { slug: 'ground-beef', unit: 'lb', item: 'Ground beef, 100% beef', perPack: 1 },
++ APU0000702111: { slug: 'bread-white', unit: 'lb', item: 'Bread, white, pan', perPack: 1.25 },
++ APU0000701312: { slug: 'white-rice', unit: 'lb', item: 'Rice, white, long grain, uncooked', perPack: 2 },
++ APU0000701322: { slug: 'spaghetti', unit: 'lb', item: 'Spaghetti and macaroni', perPack: 1 },
++ APU0000714233: { slug: 'dried-beans', unit: 'lb', item: 'Beans, dried, any type', perPack: 1 },
++ APU0000711211: { slug: 'bananas', unit: 'lb', item: 'Bananas', perPack: 1 },
++ APU0000711311: { slug: 'oranges-3lb', unit: 'lb', item: 'Oranges, Navel', perPack: 3 },
++ APU0000711415: { slug: 'strawberries', unit: '12 oz', item: 'Strawberries, dry pint', perPack: 16 / 12 },
++ APU0000712112: { slug: 'potatoes-5lb', unit: 'lb', item: 'Potatoes, white', perPack: 5 },
++ APU0000709112: { slug: 'whole-milk', unit: 'gallon', item: 'Milk, fresh, whole, fortified', perPack: 1 },
++ APU0000710212: { slug: 'cheese-block', unit: 'lb', item: 'Cheddar cheese, natural', perPack: 0.5 },
++};
++
++const text = (v, max) => (v == null || v === '' ? null : String(v).replace(/\s+/g, ' ').trim().slice(0, max) || null);
++
++/** A Sources item → a food_official_prices row, or null when it is not a series this list prices. */
++function rowOf(item) {
++ const f = (item && item.fields) || {};
++ const series = text(item && item.identity, 20);
++ if (!series || !Object.prototype.hasOwnProperty.call(SERIES, series)) return null;
++ const value = Number(f.value);
++ const year = Number(f.year);
++ const period = text(f.period, 3);
++ if (!Number.isFinite(value) || value <= 0 || value > 1000 || !Number.isInteger(year) || year < 2000 || year > 2100 || !/^M(0[1-9]|1[0-2])$/.test(period || '')) return null;
++ return {
++ series_id: series,
++ item_id: String(item.id),
++ revision: Number(item.revision) || 1,
++ value: Math.round(value * 1000) / 1000,
++ year,
++ period,
++ retrieved_at: (item.provenance && item.provenance.retrieved_at) || new Date(0).toISOString(),
++ };
++}
++
++/** What a food's page shows for one row: the series' figure and month, and the price worked out for the food's pack. */
++function shape(row) {
++ const def = SERIES[row.series_id];
++ const value = Number(row.value);
++ return {
++ slug: def.slug,
++ series_id: row.series_id,
++ item: def.item,
++ unit: def.unit,
++ value,
++ pack_price: Math.round(value * def.perPack * 100) / 100,
++ month: `${MONTHS[Number(row.period.slice(1)) - 1]} ${row.year}`,
++ url: `https://data.bls.gov/timeseries/${row.series_id}`,
++ };
++}
++
++function createOfficialPrices({ config, s, fetchImpl = globalThis.fetch, log = console, source = null }) {
++ const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');
++ const client = source || (url && config.oauth.clientSecret
++ ? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'food' } }, fetchImpl })
++ : null);
++ const cursor = ingest.createChangeCursor(s.db, { prefix: 'food', now: s.now });
++ const stats = { pulls: 0, applied: 0, removed: 0, last_at: null, last_error: null };
++ let byFood = new Map(); // food slug → shape(row), refreshed after each pull
++ let timer = null;
++ let first = null;
++ let running = false;
++
++ async function apply(item, t) {
++ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'price') return 'hold';
++ if (item.removed) {
++ await t.query('DELETE FROM food_official_prices WHERE item_id = $1', [String(item.id)]);
++ return 'removed';
++ }
++ const row = rowOf(item);
++ if (!row) return 'hold';
++ const cols = Object.keys(row);
++ // A replayed page (or an older revision arriving late) never takes a row back.
++ await t.query(`INSERT INTO food_official_prices (${cols.join(', ')}) VALUES (${cols.map((_, i) => `$${i + 1}`).join(', ')})
++ ON CONFLICT (series_id) DO UPDATE SET ${cols.filter((c) => c !== 'series_id').map((c) => `${c} = EXCLUDED.${c}`).join(', ')}
++ WHERE food_official_prices.item_id <> EXCLUDED.item_id OR food_official_prices.revision <= EXCLUDED.revision`, cols.map((c) => row[c]));
++ return 'applied';
++ }
++
++ async function load() {
++ const rows = await s.db.many('SELECT series_id, value, year, period FROM food_official_prices');
++ byFood = new Map(rows.filter((r) => SERIES[r.series_id]).map((r) => [SERIES[r.series_id].slug, shape(r)]));
++ return byFood.size;
++ }
++
++ /** Follow Sources' change feed to its end. Never throws: the caller is a timer or a test. */
++ async function pull() {
++ if (!client || running) return { skipped: true };
++ running = true;
++ try {
++ const out = await ingest.pullChanges({ db: s.db, cursor, source: client, name: CURSOR, apply });
++ stats.pulls++;
++ stats.applied += out.applied;
++ stats.removed += out.removed;
++ stats.last_at = s.iso();
++ stats.last_error = null;
++ await load();
++ if (out.applied || out.removed) log.log(`[Prices] ${out.applied} BLS series updated, ${out.removed} removed (Sources cursor ${out.after})`);
++ return out;
++ } catch (err) {
++ stats.last_error = text(err && err.message, 300);
++ log.warn(`[Prices] pull from OpenVibe.Sources failed: ${stats.last_error}`);
++ return { error: stats.last_error };
++ } finally { running = false; }
++ }
++
++ return {
++ enabled: Boolean(client),
++ stats,
++ pull,
++ apply,
++ load,
++ /** The BLS figure for a food on the list, or null when BLS prices nothing like it (or nothing is read yet). */
++ forFood(slug) { return byFood.get(slug) || null; },
++ start() {
++ load().catch(() => { /* the first pull loads it */ });
++ if (!client || timer) return;
++ first = setTimeout(() => { pull(); }, FIRST_MS);
++ timer = setInterval(() => { pull(); }, EVERY_MS);
++ if (first.unref) first.unref();
++ if (timer.unref) timer.unref();
++ },
++ stop() {
++ if (first) clearTimeout(first);
++ if (timer) clearInterval(timer);
++ first = null;
++ timer = null;
++ },
++ };
++}
++
++module.exports = { createOfficialPrices, rowOf, shape, SERIES, SOURCE_KEY, NOTE };
ddiff --git a/server/http/pages.js b/server/http/pages.js
iindex 23e395e..a2b17d2 100644
---- a/server/http/pages.js
++++ b/server/http/pages.js
@@@ -28,6 +28,7 @@ const { html, raw, table, notice, time, badge } = require('../render/html');
const { markdown } = require('../render/markdown');
const { send } = require('../render/layout');
const foods = require('../food/foods');
++const { NOTE: OFFICIAL_NOTE } = require('../food/official-prices');
const { planMeals } = require('../food/planner');
const { searchPlaces, safeUrl, ATTRIBUTION } = require('../food/places');
const { matchPantry } = require('../food/pantry-match');
@@@ -561,6 +562,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'
r.get('/foods/:slug', (req, res) => {
const f = foods.getFood(req.params.slug);
++ const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;
if (!f) return page(req, res, {
title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],
body: html`<h1>No such food</h1><p>Nothing in the list is called “${String(req.params.slug).slice(0, 64)}”. <a href="/foods">See the whole list</a>.</p>`,
@@@ -584,6 +586,9 @@ ${table(['', 'A serving', f.per_100g ? 'Per 100 g' : 'Per 100 g'], [
<p>${f.price == null ? 'No price recorded.' : html`<b>${money(f.price)}</b> to ${money(f.price_max)} a pack · ${f.band_label} · ${money(f.cost_per_serving)} a serving`}</p>
${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)]))}
<p class="muted small">Typical 2025 US (Washington) prices carried over from the OpenVibe.Tools grocery dataset; not a live price.</p>
++${official ? html`<h2>US average price</h2>
++<p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.toLowerCase()} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p>
++<p class="muted small">${OFFICIAL_NOTE}</p>` : ''}
${f.tags.length ? html`<p class="tags">${f.tags.map((t) => badge(t))}</p>` : ''}
<p><a href="/plan?people=2&days=3">Plan meals with this</a> · <a href="/foods">Back to the list</a></p>`,
});
ddiff --git a/server/index.js b/server/index.js
iindex ee8a5d5..d876534 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -44,9 +44,11 @@ async function start() {
// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.
ctx.library.start();
ctx.recipeSearch.start();
-- // Food recalls follow OpenVibe.Sources' change feed hourly (server/food/recalls.js).
++ // Food recalls and BLS's average prices follow OpenVibe.Sources' change feed hourly (server/food/recalls.js,
++ // server/food/official-prices.js).
ctx.recalls.start();
-- const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop()];
++ ctx.officialPrices.start();
++ const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop()];
createLifecycle({ server, ctx, extra });
return { server, ctx };
}
ddiff --git a/test/helpers/boot.js b/test/helpers/boot.js
iindex 654b209..1ca2519 100644
---- a/test/helpers/boot.js
++++ b/test/helpers/boot.js
@@@ -48,7 +48,7 @@ async function boot(opts = {}) {
const { createStore } = require('../../server/db');
const testdb = await require('./db').testDb();
const store = createStore(testdb.db, { now: opts.now });
-- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource });
++ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource });
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
ddiff --git a/test/official-prices.test.js b/test/official-prices.test.js
nnew file mode 100644
iindex 0000000..81c47dc
---- /dev/null
++++ b/test/official-prices.test.js
@@@ -0,0 +1,90 @@
++'use strict';
++/**
++ * Official prices (server/food/official-prices.js): BLS's US city average prices from OpenVibe.Sources.
++ * - a bls-average-prices item for a series on the list becomes one row; another series, kind or source is held;
++ * - the pull follows the change feed from its own cursor (the recalls keep theirs), a removed item is deleted, an
++ * older revision never wins;
++ * - a food's page shows the figure, its month, the price worked out for its pack and BLS's note; a food BLS does not
++ * price shows none.
++ */
++const assert = require('assert');
++const { boot, check, done } = require('./helpers/boot');
++const foods = require('../server/food/foods');
++const { rowOf, shape, SERIES } = require('../server/food/official-prices');
++
++const item = (series, n, over = {}, fields = {}) => ({
++ id: `itm_p${n}`, source_key: 'bls-average-prices', category: 'food', kind: 'price', identity: series, canonical_url: null, title: series, summary: null,
++ fields: { value: '2.272', year: '2026', period: 'M09', ...fields },
++ revision: 1, provenance: { retrieved_at: '2026-10-24T06:00:00.000Z' }, removed: null, change_seq: n, ...over,
++});
++
++function feed(items) {
++ return {
++ items,
++ async listItems({ after = 0, limit = 100 }) {
++ const rest = items.filter((x) => x.change_seq > after).sort((a, b) => a.change_seq - b.change_seq);
++ const page = rest.slice(0, limit);
++ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: rest.length > limit, sources: {} };
++ },
++ };
++}
++
++(async () => {
++ const items = [
++ item('APU0000708111', 1),
++ item('APU0000702111', 2, {}, { value: '1.830' }),
++ item('APU0000711415', 3, {}, { value: '2.840' }),
++ item('APU0000799999', 4), // a series the list does not price
++ { ...item('APU0000709112', 5), kind: 'record' }, // another kind
++ { ...item('APU0000709112', 6), source_key: 'openfda-food-enforcement' },
++ ];
++ const source = feed(items);
++ const t = await boot({ priceSource: source });
++ try {
++ await check('every series prices a food on the list; a row is the item as BLS states it', async () => {
++ for (const [series, def] of Object.entries(SERIES)) assert.ok(foods.getFood(def.slug), `${series} → ${def.slug}`);
++ const row = rowOf(item('APU0000708111', 1));
++ assert.deepStrictEqual([row.series_id, row.value, row.year, row.period], ['APU0000708111', 2.272, 2026, 'M09']);
++ assert.strictEqual(rowOf(item('APU0000799999', 1)), null, 'a series nobody maps');
++ assert.strictEqual(rowOf(item('APU0000708111', 1, {}, { value: '-1' })), null);
++ assert.strictEqual(rowOf(item('APU0000708111', 1, {}, { period: 'M13' })), null, 'M13 is BLS\'s annual average, not a month');
++ const bread = shape({ series_id: 'APU0000702111', value: '1.830', year: 2026, period: 'M09' });
++ assert.deepStrictEqual([bread.slug, bread.month, bread.pack_price, bread.unit], ['bread-white', 'September 2026', 2.29, 'lb'], 'a 20 oz loaf is 1.25 lb');
++ assert.strictEqual(shape({ series_id: 'APU0000711415', value: '2.840', year: 2026, period: 'M09' }).pack_price, 3.79, 'strawberries: 12 oz → 1 lb');
++ });
++
++ await check('the pull follows the feed on its own cursor; anything else is held', async () => {
++ assert.strictEqual((await t.get('/foods/eggs-dozen')).text.includes('US average price'), false, 'nothing before the first pull');
++ const out = await t.ctx.officialPrices.pull();
++ assert.deepStrictEqual([out.applied, out.hold], [3, 3]);
++ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.prices.after'")), 6);
++ assert.strictEqual(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.after'"), null, 'the recalls\' cursor is theirs');
++ assert.strictEqual(t.ctx.officialPrices.forFood('eggs-dozen').value, 2.272);
++ });
++
++ await check('a new month replaces the old; an older revision arriving late and a removed item behave', async () => {
++ source.items.push(item('APU0000708111', 7, { id: 'itm_p1', revision: 3 }, { value: '2.301', period: 'M10' })); // the same Sources item, a new month
++ source.items.push(item('APU0000708111', 8, { id: 'itm_p1', revision: 2 }, { value: '9.999', period: 'M08' })); // a replayed older revision
++ source.items.push({ ...item('APU0000711415', 9), id: 'itm_p3', removed: { at: '2026-10-24T07:00:00.000Z', reason: 'licence' }, revision: 2 });
++ const out = await t.ctx.officialPrices.pull();
++ assert.strictEqual(out.removed, 1);
++ const eggs = t.ctx.officialPrices.forFood('eggs-dozen');
++ assert.deepStrictEqual([eggs.value, eggs.month], [2.301, 'October 2026']);
++ assert.strictEqual(t.ctx.officialPrices.forFood('strawberries'), null);
++ });
++
++ await check('a food\'s page shows the BLS figure for its pack with the note; one BLS does not price shows none', async () => {
++ const bread = await t.get('/foods/bread-white');
++ assert.strictEqual(bread.status, 200);
++ assert.ok(bread.text.includes('US average price'));
++ assert.ok(bread.text.includes('$2.29</b> for this pack at the US city average for September 2026: $1.83 per lb'), bread.text.slice(bread.text.indexOf('US average'), bread.text.indexOf('US average') + 300));
++ assert.ok(bread.text.includes('https://data.bls.gov/timeseries/APU0000702111'));
++ assert.ok(bread.text.includes('BLS.gov cannot vouch for the data'));
++ const tuna = await t.get('/foods/canned-tuna');
++ assert.ok(!tuna.text.includes('US average price'));
++ });
++ } finally {
++ await t.close();
++ }
++ done();
++})();