openvibers/openvibe.food Commits
Food: Open Food Facts labels beside the food list, credited ODbL (#149)
Changes
@@@ -61,6 +61,10 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by- `food_recalls`, `food_ingest_cursor` (0009) — the FDA's food recall reports as OpenVibe.Sources holds them, one rowper recall number, and the cursor in Sources' change feed. Public domain data; nothing about a person.++- `food_off_products` (0011) — Open Food Facts products matched to the list, one row per product code with the label++ figures, the food slug they label and the Sources item and revision they came from. ODbL data, credited on the food++ page; nothing about a person ([migrations/0011_open_food_facts.sql](migrations/0011_open_food_facts.sql)).++0001 creates no tables. The service is the authority for these rows and for nothing else.## Does not own@@@ -163,6 +167,12 @@ published average-price file once a day (source `bls-average-prices`) and Food f`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list'sown band stays as it is.++**Open Food Facts labels** — Open Food Facts' daily product export (ODbL), read by OpenVibe.Sources as a streamed++bulk dataset (source `open-food-facts`), is matched by product name to the 42 foods on the list. A food's page shows++the most complete matching product's per-100 g figures, its Nutri-Score and a link to the product, in a separate++section credited "Open Food Facts (ODbL)" ([server/food/off.js](server/food/off.js), `food_off_products`), beside++the list's own nutrition and prices and never replacing them.++**Nutrition** — the same source's typical values per serving. "Per 100 g" is arithmetic on the pack size stated in anitem's own name divided by its servings (for example a 15 oz can with 3.5 servings is 121 g a serving). Where a namestates no weight — a dozen eggs, a 12-pack of ramen — no per-100 g figure is given rather than a density guessed.nnew file mode 100644@@@ -0,0 +1,28 @@++-- phase: expand++-- OpenVibe.Food: Open Food Facts products (source open-food-facts, category food) as label data beside the food list++-- (server/food/off.js). Sources streams the ODbL export and keeps each product's code, name, brands, Nutri-Score and++-- per-100 g nutrients; Food follows Sources' change feed from its food_ingest_cursor and keeps the products that name++-- one of the 42 foods it plans with. The list's own nutrition and typical prices stay in server/food/foods.js and are++-- never replaced. Open Food Facts' ODbL credit and a link to the product are shown wherever these figures appear.++++CREATE TABLE food_off_products (++ code text COLLATE "C" PRIMARY KEY, -- Open Food Facts' product code, Sources' identity++ item_id text COLLATE "C" NOT NULL UNIQUE, -- the Sources item++ revision integer NOT NULL, -- the Sources item's revision this row holds++ slug text COLLATE "C" NOT NULL, -- the food on the list it labels (server/food/foods.js)++ product_name text NOT NULL, -- product_name, as the label states it++ brands text, -- brands++ product_url text, -- the Open Food Facts product page++ nutriscore_grade text, -- Nutri-Score a-e, upper case, when the product states one++ energy_kcal_100g double precision,++ fat_100g double precision,++ saturated_fat_g double precision,++ carbs_g double precision,++ sugars_g double precision,++ protein_g double precision,++ salt_g double precision,++ fiber_g double precision,++ retrieved_at text COLLATE "C" NOT NULL -- when Sources last read it, ISO 8601 UTC++);++++CREATE INDEX food_off_products_slug ON food_off_products (slug);@@@ -98,6 +98,8 @@ async function createApp(opts = {}) {ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });++ // Open Food Facts products as label data beside the list, from OpenVibe.Sources (server/food/off.js); opts.offSource.++ ctx.off = require('./food/off').createOff({ config, s, fetchImpl, log, source: opts.offSource || null });const app = express();app.disable('x-powered-by');nnew file mode 100644@@@ -0,0 +1,237 @@++'use strict';++++/**++ * Open Food Facts products as label data beside the food list (source open-food-facts, category food).++ *++ * Sources streams Open Food Facts' export (ODbL) and keeps each product's code, name, brands, Nutri-Score and++ * per-100 g nutrients. Food follows Sources' change feed from its own cursor, matches a product to one of the++ * 42 foods it plans with by the product's own name, and keeps the match in food_off_products. The list's own++ * nutrition and typical prices stay what they are: a food's page shows the product's figures in a separate++ * section, as the label states them, with the ODbL credit and a link to the product — never as a replacement.++ *++ * Pulls 50 seconds after start and then every six hours. Off without OV_SOURCES_INTERNAL_URL and the OAuth++ * client secret (a test passes its own source).++ */++const ingest = require('openvibe-publishing/ingest');++++const SOURCE_KEY = 'open-food-facts';++const CURSOR = 'sources.food.off.after';++const EVERY_MS = 6 * 60 * 60 * 1000;++const FIRST_MS = 50 * 1000;++const ODBL_URL = 'https://opendatacommons.org/licenses/odbl/1-0/';++const NOTE = 'Product label data from Open Food Facts (ODbL). The Open Food Facts database is available under the Open Database License; attribution and share-alike apply.';++++const text = (v, max) => (v == null || v === '' ? null : String(v).replace(/\s+/g, ' ').trim().slice(0, max) || null);++const num = (v) => {++ if (v == null || v === '') return null;++ const n = typeof v === 'number' ? v : Number(String(v).replace(',', '.'));++ return Number.isFinite(n) && n >= 0 && n < 1_000_000 ? Math.round(n * 1000) / 1000 : null;++};++++/** The food list's own matchers, in the order that keeps "peanut butter" from becoming butter. */++const MATCHERS = [++ ['peanut-butter', /peanut butter/],++ ['whole-milk', /\bwhole milk\b/, /\b(milk chocolate|chocolate milk|milk powder|powdered milk|coconut milk|almond milk|soy milk|oat milk|whole milk powder)\b/],++ ['powdered-milk', /\b(powdered milk|milk powder|nonfat dry milk)\b/],++ ['cheese-block', /\b(cheddar|cheese block|block cheese)\b/, /\b(shredded|sliced|sauce|spread|puffs|snack)\b/],++ ['yogurt', /\b(yogurt|yoghurt)\b/],++ ['butter', /\bbutter\b/, /\b(peanut|almond|cashew|cocoa|apple|shea|body)\b/],++ ['eggs-dozen', /\beggs?\b/, /\b(egg noodles?|egg rolls?|candy|easter)\b/],++ ['chicken-breast', /\bchicken breast\b/],++ ['ground-beef', /\bground beef\b/],++ ['hot-dogs', /\b(hot dogs?|frankfurters?|wieners?)\b/],++ ['canned-tuna', /\btuna\b/, /\b(tuna casserole|tuna salad)\b/],++ ['sardines', /\bsardines?\b/],++ ['canned-chili', /\bchili\b/, /\b(chili powder|chili sauce|chili oil|chili flakes?)\b/],++ ['canned-soup', /\b(chicken noodle soup|condensed soup|soup)\b/, /\b(soup mix|soup base|soup bowl)\b/],++ ['ramen-12pk', /\b(ramen|instant noodles?)\b/],++ ['mac-cheese', /\b(mac(aroni)? (and|&) cheese|macaroni cheese)\b/],++ ['spaghetti', /\b(spaghetti|thin spaghetti)\b/],++ ['white-rice', /\b(long grain white rice|white rice)\b/],++ ['oatmeal', /\b(oatmeal|old fashioned oats|rolled oats|quick oats)\b/, /\b(oat milk|oat cereal|granola)\b/],++ ['bread-white', /\b(white bread|white sandwich bread)\b/],++ ['tortillas', /\btortillas?\b/],++ ['jiffy-mix', /\b(corn muffin mix|muffin mix)\b/],++ ['granola-bars', /\bgranola bars?\b/],++ ['instant-coffee', /\binstant coffee\b/],++ ['canned-beans', /\b(canned|black|kidney|pinto|white) beans\b/, /\b(dried|dry|green beans?|string beans?|bean sprouts?)\b/],++ ['dried-beans', /\b(dried|dry) (pinto|black|kidney|white|great northern) beans\b/],++ ['canned-corn', /\bcorn\b/, /\b(corn flakes?|corn chips?|corn starch|corn syrup|popcorn|cornmeal|corn dogs?)\b/],++ ['baby-carrots', /\bbaby carrots?\b/],++ ['canned-green-beans', /\bgreen beans?\b/],++ ['frozen-mixed-veg', /\b(mixed vegetables?|frozen mixed veg)\b/],++ ['onion-3lb', /\bonions?\b/, /\b(onion powder|onion rings?|onion soup)\b/],++ ['canned-tomatoes', /\b(diced|crushed|whole) tomatoes?\b/],++ ['potatoes-5lb', /\bpotatoes?\b/, /\b(chips?|crisps?|fries|flakes?|starch|salad)\b/],++ ['bananas', /\bbananas?\b/, /\b(banana chips?|banana bread)\b/],++ ['apples-3lb', /\bapples?\b/, /\b(apple juice|apple sauce|applesauce|apple cider|apple vinegar|pineapple)\b/],++ ['oranges-3lb', /\boranges?\b/, /\b(orange juice|orange soda|orange chicken)\b/],++ ['canned-fruit', /\b(peaches|fruit cocktail|canned fruit)\b/],++ ['raisins', /\braisins?\b/],++ ['strawberries', /\bstrawberries\b/],++ ['cooking-oil', /\b(vegetable oil|canola oil|sunflower oil)\b/],++ ['avocado', /\bavocados?\b/, /\b(oil|guacamole)\b/],++ ['trail-mix', /\b(trail mix|mixed nuts)\b/],++];++++/** A product's own name → the slug on the food list it labels, or null when the list has no such food. */++function matchSlug(name) {++ const s = String(name == null ? '' : name).normalize('NFKD').replace(/[\u0300-\u036f]/g, '')++ .toLowerCase().replace(/&/g, ' and ').replace(/[^a-z0-9]+/g, ' ').trim();++ if (!s) return null;++ for (const [slug, yes, no] of MATCHERS) {++ if (yes.test(s) && !(no && no.test(s))) return slug;++ }++ return null;++}++++// food_off_products columns -> the Sources field they hold (the header has a '-' in two of them).++const NUTRIENTS = {++ energy_kcal_100g: 'energy_kcal_100g',++ fat_100g: 'fat_100g',++ saturated_fat_g: 'saturated_fat_100g',++ carbs_g: 'carbohydrates_100g',++ sugars_g: 'sugars_100g',++ protein_g: 'proteins_100g',++ salt_g: 'salt_100g',++ fiber_g: 'fiber_100g',++};++++/** A Sources item → the food_off_products row it states, or null when it is not a product this list has. */++function rowOf(item) {++ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'food_product') return null;++ const code = String(item.identity == null ? '' : item.identity).trim();++ if (!/^\d{6,14}$/.test(code)) return null;++ const name = text(item.title, 300);++ if (!name) return null;++ const slug = matchSlug(name);++ if (!slug) return null;++ const f = item.fields || {};++ const grade = text(f.nutriscore_grade, 3);++ const row = {++ code,++ item_id: String(item.id || ''),++ revision: Number(item.revision) || 1,++ slug,++ product_name: name,++ brands: text(f.brands, 200),++ product_url: item.canonical_url || `https://world.openfoodfacts.org/product/${code}`,++ nutriscore_grade: grade && /^[a-e]$/i.test(grade) ? grade.toUpperCase() : null,++ retrieved_at: (item.provenance && item.provenance.retrieved_at) || new Date(0).toISOString(),++ };++ for (const [col, field] of Object.entries(NUTRIENTS)) row[col] = num(f[field]);++ if (Object.keys(NUTRIENTS).every((col) => row[col] == null) && !row.nutriscore_grade) return null;++ return row;++}++++/** The shape a food's page shows: the product as its label states it, per 100 g. */++function shape(row) {++ return {++ code: row.code,++ name: row.product_name,++ brands: row.brands || null,++ url: row.product_url || `https://world.openfoodfacts.org/product/${row.code}`,++ nutriscore_grade: row.nutriscore_grade || null,++ per_100g: {++ energy_kcal: row.energy_kcal_100g == null ? null : Number(row.energy_kcal_100g),++ fat_g: row.fat_100g == null ? null : Number(row.fat_100g),++ saturated_fat_g: row.saturated_fat_g == null ? null : Number(row.saturated_fat_g),++ carbs_g: row.carbs_g == null ? null : Number(row.carbs_g),++ sugars_g: row.sugars_g == null ? null : Number(row.sugars_g),++ protein_g: row.protein_g == null ? null : Number(row.protein_g),++ salt_g: row.salt_g == null ? null : Number(row.salt_g),++ fiber_g: row.fiber_g == null ? null : Number(row.fiber_g),++ },++ retrieved_at: row.retrieved_at,++ note: NOTE,++ license_url: ODBL_URL,++ };++}++++function createOff({ config, s, fetchImpl = globalThis.fetch, log = console, source = null }) {++ const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');++ const client = source || (url && config.oauth.clientSecret++ ? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'food' } }, fetchImpl })++ : null);++ const cursor = ingest.createChangeCursor(s.db, { prefix: 'food', now: s.now });++ const stats = { pulls: 0, applied: 0, removed: 0, last_at: null, last_error: null };++ let byFood = new Map(); // food slug → shape(row), the most complete product for it++ let timer = null;++ let first = null;++ let running = false;++++ async function apply(item, t) {++ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'food_product') return 'hold';++ if (item.removed) {++ await t.query('DELETE FROM food_off_products WHERE item_id = $1', [String(item.id)]);++ return 'removed';++ }++ const row = rowOf(item);++ if (!row) return 'hold';++ const cols = Object.keys(row);++ await t.query(`INSERT INTO food_off_products (${cols.join(', ')}) VALUES (${cols.map((_, i) => `$${i + 1}`).join(', ')})++ ON CONFLICT (code) DO UPDATE SET ${cols.filter((c) => c !== 'code').map((c) => `${c} = EXCLUDED.${c}`).join(', ')}++ WHERE food_off_products.item_id <> EXCLUDED.item_id OR food_off_products.revision <= EXCLUDED.revision`, cols.map((c) => row[c]));++ return 'applied';++ }++++ /** One product per food: the one that states the most nutrient values (then its Nutri-Score). */++ async function load() {++ const rows = await s.db.many('SELECT * FROM food_off_products');++ const best = new Map();++ const score = (r) => Object.keys(NUTRIENTS).filter((col) => r[col] != null).length + (r.nutriscore_grade ? 1 : 0);++ for (const row of rows) {++ const cur = best.get(row.slug);++ if (!cur || score(row) > score(cur) || (score(row) === score(cur) && row.code < cur.code)) best.set(row.slug, row);++ }++ byFood = new Map([...best.entries()].map(([slug, row]) => [slug, shape(row)]));++ return byFood.size;++ }++++ /** Follow Sources' change feed to its end. Never throws: the caller is a timer or a test. */++ async function pull() {++ if (!client || running) return { skipped: true };++ running = true;++ try {++ const out = await ingest.pullChanges({ db: s.db, cursor, source: client, name: CURSOR, apply });++ stats.pulls++;++ stats.applied += out.applied;++ stats.removed += out.removed;++ stats.last_at = s.iso();++ stats.last_error = null;++ await load();++ if (out.applied || out.removed) log.log(`[OFF] ${out.applied} Open Food Facts products updated, ${out.removed} removed (Sources cursor ${out.after})`);++ return out;++ } catch (err) {++ stats.last_error = text(err && err.message, 300);++ log.warn(`[OFF] pull from OpenVibe.Sources failed: ${stats.last_error}`);++ return { error: stats.last_error };++ } finally { running = false; }++ }++++ return {++ enabled: Boolean(client),++ stats,++ pull,++ apply,++ load,++ /** The Open Food Facts product for a food on the list, or null when no product matched (or nothing is read yet). */++ forFood(slug) { return byFood.get(slug) || null; },++ start() {++ load().catch(() => { /* the first pull loads it */ });++ if (!client || timer) return;++ first = setTimeout(() => { pull(); }, FIRST_MS);++ timer = setInterval(() => { pull(); }, EVERY_MS);++ if (first.unref) first.unref();++ if (timer.unref) timer.unref();++ },++ stop() {++ if (first) clearTimeout(first);++ if (timer) clearInterval(timer);++ first = null;++ timer = null;++ },++ };++}++++module.exports = { createOff, rowOf, shape, matchSlug, MATCHERS, SOURCE_KEY, NOTE, ODBL_URL };@@@ -597,6 +597,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'r.get('/foods/:slug', (req, res) => {const f = foods.getFood(req.params.slug);const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;++ const off = f && ctx.off ? ctx.off.forFood(f.slug) : null;if (!f) return page(req, res, {title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],body: html`<h1>No such food</h1><p>Nothing in the list is called “${String(req.params.slug).slice(0, 64)}”. <a href="/foods">See the whole list</a>.</p>`,@@@ -623,6 +624,19 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])${official ? html`<h2>US average price</h2><p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p><p class="muted small">${OFFICIAL_NOTE}</p>` : ''}++${off ? html`<h2>Open Food Facts label</h2>++<p><b>${off.name}</b>${off.brands ? html` · ${off.brands}` : ''}${off.nutriscore_grade ? html` · Nutri-Score ${off.nutriscore_grade}` : ''}</p>++${table(['Per 100 g', 'As the label states'], [++ ['Energy', off.per_100g.energy_kcal == null ? '—' : `${off.per_100g.energy_kcal} kcal`],++ ['Fat', off.per_100g.fat_g == null ? '—' : `${off.per_100g.fat_g} g`],++ ['of which saturates', off.per_100g.saturated_fat_g == null ? '—' : `${off.per_100g.saturated_fat_g} g`],++ ['Carbohydrate', off.per_100g.carbs_g == null ? '—' : `${off.per_100g.carbs_g} g`],++ ['of which sugars', off.per_100g.sugars_g == null ? '—' : `${off.per_100g.sugars_g} g`],++ ['Protein', off.per_100g.protein_g == null ? '—' : `${off.per_100g.protein_g} g`],++ ['Salt', off.per_100g.salt_g == null ? '—' : `${off.per_100g.salt_g} g`],++ ['Fibre', off.per_100g.fiber_g == null ? '—' : `${off.per_100g.fiber_g} g`],++ ])}++<p class="muted small"><a href="${off.url}" rel="noopener">Open Food Facts (ODbL)</a> · product ${off.code}${off.retrieved_at ? html`, read ${time(off.retrieved_at)}` : ''}. ${off.note} These are the product's own label figures, shown beside the typical values above and not replacing them.</p>` : ''}${f.tags.length ? html`<p class="tags">${f.tags.map((t) => badge(t))}</p>` : ''}<p><a href="/plan?people=2&days=3">Plan meals with this</a> · <a href="/foods">Back to the list</a></p>`,});@@@ -44,13 +44,14 @@ async function start() {// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.ctx.library.start();ctx.recipeSearch.start();-- // Food recalls and BLS's average prices follow OpenVibe.Sources' change feed hourly (server/food/recalls.js,-- // server/food/official-prices.js).++ // Food recalls, BLS's average prices and Open Food Facts products follow OpenVibe.Sources' change feed++ // (server/food/recalls.js, server/food/official-prices.js, server/food/off.js).ctx.recalls.start();ctx.officialPrices.start();++ ctx.off.start();// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).ctx.nearWarm.start();-- const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.nearWarm.stop()];++ const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];createLifecycle({ server, ctx, extra });return { server, ctx };}@@@ -50,7 +50,7 @@ async function boot(opts = {}) {const { createStore } = require('../../server/db');const testdb = await require('./db').testDb();const store = createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource });++ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, offSource: opts.offSource });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;nnew file mode 100644@@@ -0,0 +1,109 @@++'use strict';++/**++ * Open Food Facts (server/food/off.js): products from OpenVibe.Sources as label data beside the food list.++ * - a product's own name maps to one of the list's foods; chips, juice and flakes do not become the food;++ * - the pull follows the change feed from its own cursor and keeps the most complete product for a food;++ * - a food's page shows the product's per-100 g figures with the Open Food Facts (ODbL) credit and link, beside++ * the list's own nutrition and prices, never instead of them;++ * - the match survives a restart because it is in food_off_products.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { rowOf, matchSlug, shape, NOTE } = require('../server/food/off');++const { createOff } = require('../server/food/off');++++const item = (code, title, fields = {}, over = {}) => ({++ id: `itm_off_${code}`, source_key: 'open-food-facts', category: 'food', kind: 'food_product', identity: code,++ canonical_url: `https://world.openfoodfacts.org/product/${code}`, title, summary: null,++ fields: { brands: 'Example Brand', nutriscore_grade: 'b', ...fields },++ revision: 1, provenance: { retrieved_at: '2026-10-30T06:00:00.000Z' }, removed: null, ...over,++});++++function feed(items) {++ return {++ items,++ async listItems({ after = 0, limit = 100 }) {++ const rest = items.filter((x) => x.change_seq > after).sort((a, b) => a.change_seq - b.change_seq);++ const page = rest.slice(0, limit);++ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: rest.length > limit, sources: {} };++ },++ };++}++++(async () => {++ const items = [++ { ...item('0000000000011', 'Large Eggs', { energy_kcal_100g: '143', proteins_100g: '12.6' }), change_seq: 1 },++ { ...item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '140', proteins_100g: '13', fat_100g: '9.5', carbohydrates_100g: '0.7', salt_100g: '0.3' }, { revision: 3 }), change_seq: 2 },++ { ...item('0000000000021', 'Whole Milk', { energy_kcal_100g: '64', proteins_100g: '3.3', fat_100g: '3.6', carbohydrates_100g: '4.8', sugars_100g: '4.8' }), change_seq: 3 },++ { ...item('0000000000031', 'Banana Chips', { energy_kcal_100g: '519' }), change_seq: 4 }, // a snack, not a banana++ { ...item('0000000000041', 'Potato Chips', { energy_kcal_100g: '536' }), change_seq: 5 }, // not the potato on the list++ { ...item('0000000000051', 'Apple Juice', { energy_kcal_100g: '46' }), change_seq: 6 }, // not an apple++ { ...item('0000000000061', 'Mystery Product', {}), change_seq: 7 }, // no figures, no match++ { ...item('0000000000071', 'Whole Milk', { proteins_100g: '3.3' }, { kind: 'record' }), change_seq: 8 },++ { ...item('0000000000081', 'Whole Milk', { proteins_100g: '3.3' }, { source_key: 'openfda-food-enforcement' }), change_seq: 9 },++ ];++ const source = feed(items);++ const t = await boot({ offSource: source });++ try {++ await check('names map to the list\'s foods, and a snack, a juice or a chip does not', async () => {++ assert.strictEqual(matchSlug('Peanut Butter, Creamy'), 'peanut-butter', 'peanut butter is not butter');++ assert.strictEqual(matchSlug('Whole Milk'), 'whole-milk');++ assert.strictEqual(matchSlug('Cheddar Cheese Block'), 'cheese-block');++ assert.strictEqual(matchSlug('Dried Pinto Beans'), 'dried-beans');++ assert.strictEqual(matchSlug('Canned Black Beans'), 'canned-beans');++ assert.strictEqual(matchSlug('Banana Chips'), null);++ assert.strictEqual(matchSlug('Potato Chips'), null);++ assert.strictEqual(matchSlug('Apple Juice'), null);++ assert.strictEqual(matchSlug('Corn Flakes'), null);++ const row = rowOf(item('0000000000021', 'Whole Milk', { energy_kcal_100g: '64', proteins_100g: '3.3' }));++ assert.deepStrictEqual([row.slug, row.product_name, row.energy_kcal_100g, row.protein_g], ['whole-milk', 'Whole Milk', 64, 3.3]);++ assert.strictEqual(rowOf(item('0000000000061', 'Mystery Product', {})), null, 'no nutrient and no Nutri-Score = nothing to show');++ assert.strictEqual(rowOf(item('abc', 'Whole Milk')), null, 'a code that is not a product code');++ });++++ await check('the pull follows the feed: matches applied, everything else held', async () => {++ const out = await t.ctx.off.pull();++ assert.deepStrictEqual([out.applied, out.hold], [3, 6], 'eggs twice, milk once; the snacks and the two wrong items held');++ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.off.after'")), 9);++ const eggs = t.ctx.off.forFood('eggs-dozen');++ assert.strictEqual(eggs.code, '0000000000012', 'the more complete product is the one shown');++ assert.deepStrictEqual([eggs.name, eggs.brands, eggs.nutriscore_grade], ['Free Range Large Eggs', 'Example Brand', 'B']);++ assert.deepStrictEqual([eggs.per_100g.energy_kcal, eggs.per_100g.protein_g, eggs.per_100g.salt_g], [140, 13, 0.3]);++ assert.strictEqual(t.ctx.off.forFood('canned-tuna'), null, 'a food no product matched shows none');++ });++++ await check('a food\'s page shows the product with the ODbL credit, beside its own nutrition', async () => {++ const page = await t.get('/foods/eggs-dozen');++ assert.strictEqual(page.status, 200);++ assert.ok(page.text.includes('Open Food Facts label'));++ assert.ok(page.text.includes('Open Food Facts (ODbL)'), 'the exact credit');++ assert.ok(page.text.includes('https://world.openfoodfacts.org/product/0000000000012'), 'the product is linked');++ assert.ok(page.text.includes('Free Range Large Eggs'));++ assert.ok(page.text.includes('140 kcal') && page.text.includes('13 g'), 'the label figures');++ assert.ok(page.text.includes('Nutrition') && page.text.includes('A serving'), 'the list\'s own section is still there');++ assert.ok(page.text.includes(NOTE.split('.')[0]), 'the ODbL note');++ const tuna = await t.get('/foods/canned-tuna');++ assert.ok(!tuna.text.includes('Open Food Facts label'), 'a food with no match shows no label section');++ });++++ await check('the match is durable: a new instance loads it from food_off_products', async () => {++ const again = createOff({ config: { sources: { internalUrl: '' } }, s: t.ctx.s, log: { log() {}, warn() {} } });++ assert.strictEqual(await again.load(), 2);++ assert.strictEqual(again.forFood('eggs-dozen').code, '0000000000012');++ const shapeOf = shape({ ...rowOf(item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '140' })), retrieved_at: '2026-10-30T06:00:00.000Z' });++ assert.strictEqual(shapeOf.license_url, 'https://opendatacommons.org/licenses/odbl/1-0/');++ });++++ await check('a removed product is deleted; an older revision never wins', async () => {++ source.items.push({ ...item('0000000000021', 'Whole Milk', {}, { id: 'itm_off_0000000000021', removed: { at: '2026-10-30T07:00:00.000Z', reason: 'licence' }, revision: 2 }), change_seq: 10 });++ source.items.push({ ...item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '1' }, { id: 'itm_off_0000000000012', revision: 1 }), change_seq: 11 });++ const out = await t.ctx.off.pull();++ assert.strictEqual(out.removed, 1);++ assert.strictEqual(t.ctx.off.forFood('whole-milk'), null);++ assert.strictEqual(t.ctx.off.forFood('eggs-dozen').per_100g.energy_kcal, 140, 'the older revision did not overwrite the label');++ });++ } finally {++ await t.close();++ }++ done();++})().catch((err) => { console.error(err); process.exit(1); });
@@ -61,6 +61,10 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
@@ -61,6 +61,10 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
- `food_recalls`, `food_ingest_cursor` (0009) — the FDA's food recall reports as OpenVibe.Sources holds them, one row
- `food_recalls`, `food_ingest_cursor` (0009) — the FDA's food recall reports as OpenVibe.Sources holds them, one row
per recall number, and the cursor in Sources' change feed. Public domain data; nothing about a person.
per recall number, and the cursor in Sources' change feed. Public domain data; nothing about a person.
+- `food_off_products` (0011) — Open Food Facts products matched to the list, one row per product code with the label
+ figures, the food slug they label and the Sources item and revision they came from. ODbL data, credited on the food
+ page; nothing about a person ([migrations/0011_open_food_facts.sql](migrations/0011_open_food_facts.sql)).
+
0001 creates no tables. The service is the authority for these rows and for nothing else.
0001 creates no tables. The service is the authority for these rows and for nothing else.
## Does not own
## Does not own
@@ -163,6 +167,12 @@ published average-price file once a day (source `bls-average-prices`) and Food f
@@ -163,6 +167,12 @@ published average-price file once a day (source `bls-average-prices`) and Food f
`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list's
`food_official_prices`. BLS is cited with its note that it cannot vouch for data once they leave BLS.gov. The list's
own band stays as it is.
own band stays as it is.
+**Open Food Facts labels** — Open Food Facts' daily product export (ODbL), read by OpenVibe.Sources as a streamed
+bulk dataset (source `open-food-facts`), is matched by product name to the 42 foods on the list. A food's page shows
+the most complete matching product's per-100 g figures, its Nutri-Score and a link to the product, in a separate
+section credited "Open Food Facts (ODbL)" ([server/food/off.js](server/food/off.js), `food_off_products`), beside
+the list's own nutrition and prices and never replacing them.
+
**Nutrition** — the same source's typical values per serving. "Per 100 g" is arithmetic on the pack size stated in an
**Nutrition** — the same source's typical values per serving. "Per 100 g" is arithmetic on the pack size stated in an
item's own name divided by its servings (for example a 15 oz can with 3.5 servings is 121 g a serving). Where a name
item's own name divided by its servings (for example a 15 oz can with 3.5 servings is 121 g a serving). Where a name
states no weight — a dozen eggs, a 12-pack of ramen — no per-100 g figure is given rather than a density guessed.
states no weight — a dozen eggs, a 12-pack of ramen — no per-100 g figure is given rather than a density guessed.
new file mode 100644
new file mode 100644
@@ -0,0 +1,28 @@
@@ -0,0 +1,28 @@
+-- phase: expand
+-- OpenVibe.Food: Open Food Facts products (source open-food-facts, category food) as label data beside the food list
+-- (server/food/off.js). Sources streams the ODbL export and keeps each product's code, name, brands, Nutri-Score and
+-- per-100 g nutrients; Food follows Sources' change feed from its food_ingest_cursor and keeps the products that name
+-- one of the 42 foods it plans with. The list's own nutrition and typical prices stay in server/food/foods.js and are
+-- never replaced. Open Food Facts' ODbL credit and a link to the product are shown wherever these figures appear.
+
+CREATE TABLE food_off_products (
+ code text COLLATE "C" PRIMARY KEY, -- Open Food Facts' product code, Sources' identity
+ item_id text COLLATE "C" NOT NULL UNIQUE, -- the Sources item
+ revision integer NOT NULL, -- the Sources item's revision this row holds
+ slug text COLLATE "C" NOT NULL, -- the food on the list it labels (server/food/foods.js)
+ product_name text NOT NULL, -- product_name, as the label states it
+ brands text, -- brands
+ product_url text, -- the Open Food Facts product page
+ nutriscore_grade text, -- Nutri-Score a-e, upper case, when the product states one
+ energy_kcal_100g double precision,
+ fat_100g double precision,
+ saturated_fat_g double precision,
+ carbs_g double precision,
+ sugars_g double precision,
+ protein_g double precision,
+ salt_g double precision,
+ fiber_g double precision,
+ retrieved_at text COLLATE "C" NOT NULL -- when Sources last read it, ISO 8601 UTC
+);
+
+CREATE INDEX food_off_products_slug ON food_off_products (slug);
@@ -98,6 +98,8 @@ async function createApp(opts = {}) {
@@ -98,6 +98,8 @@ async function createApp(opts = {}) {
ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });
ctx.recalls = require('./food/recalls').createRecalls({ config, s, fetchImpl, log, source: opts.recallSource || null });
// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.
// BLS's US average prices for list items, from OpenVibe.Sources (server/food/official-prices.js); a test passes opts.priceSource.
ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });
ctx.officialPrices = require('./food/official-prices').createOfficialPrices({ config, s, fetchImpl, log, source: opts.priceSource || null });
+ // Open Food Facts products as label data beside the list, from OpenVibe.Sources (server/food/off.js); opts.offSource.
+ ctx.off = require('./food/off').createOff({ config, s, fetchImpl, log, source: opts.offSource || null });
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
new file mode 100644
new file mode 100644
@@ -0,0 +1,237 @@
@@ -0,0 +1,237 @@
+'use strict';
+
+/**
+ * Open Food Facts products as label data beside the food list (source open-food-facts, category food).
+ *
+ * Sources streams Open Food Facts' export (ODbL) and keeps each product's code, name, brands, Nutri-Score and
+ * per-100 g nutrients. Food follows Sources' change feed from its own cursor, matches a product to one of the
+ * 42 foods it plans with by the product's own name, and keeps the match in food_off_products. The list's own
+ * nutrition and typical prices stay what they are: a food's page shows the product's figures in a separate
+ * section, as the label states them, with the ODbL credit and a link to the product — never as a replacement.
+ *
+ * Pulls 50 seconds after start and then every six hours. Off without OV_SOURCES_INTERNAL_URL and the OAuth
+ * client secret (a test passes its own source).
+ */
+const ingest = require('openvibe-publishing/ingest');
+
+const SOURCE_KEY = 'open-food-facts';
+const CURSOR = 'sources.food.off.after';
+const EVERY_MS = 6 * 60 * 60 * 1000;
+const FIRST_MS = 50 * 1000;
+const ODBL_URL = 'https://opendatacommons.org/licenses/odbl/1-0/';
+const NOTE = 'Product label data from Open Food Facts (ODbL). The Open Food Facts database is available under the Open Database License; attribution and share-alike apply.';
+
+const text = (v, max) => (v == null || v === '' ? null : String(v).replace(/\s+/g, ' ').trim().slice(0, max) || null);
+const num = (v) => {
+ if (v == null || v === '') return null;
+ const n = typeof v === 'number' ? v : Number(String(v).replace(',', '.'));
+ return Number.isFinite(n) && n >= 0 && n < 1_000_000 ? Math.round(n * 1000) / 1000 : null;
+};
+
+/** The food list's own matchers, in the order that keeps "peanut butter" from becoming butter. */
+const MATCHERS = [
+ ['peanut-butter', /peanut butter/],
+ ['whole-milk', /\bwhole milk\b/, /\b(milk chocolate|chocolate milk|milk powder|powdered milk|coconut milk|almond milk|soy milk|oat milk|whole milk powder)\b/],
+ ['powdered-milk', /\b(powdered milk|milk powder|nonfat dry milk)\b/],
+ ['cheese-block', /\b(cheddar|cheese block|block cheese)\b/, /\b(shredded|sliced|sauce|spread|puffs|snack)\b/],
+ ['yogurt', /\b(yogurt|yoghurt)\b/],
+ ['butter', /\bbutter\b/, /\b(peanut|almond|cashew|cocoa|apple|shea|body)\b/],
+ ['eggs-dozen', /\beggs?\b/, /\b(egg noodles?|egg rolls?|candy|easter)\b/],
+ ['chicken-breast', /\bchicken breast\b/],
+ ['ground-beef', /\bground beef\b/],
+ ['hot-dogs', /\b(hot dogs?|frankfurters?|wieners?)\b/],
+ ['canned-tuna', /\btuna\b/, /\b(tuna casserole|tuna salad)\b/],
+ ['sardines', /\bsardines?\b/],
+ ['canned-chili', /\bchili\b/, /\b(chili powder|chili sauce|chili oil|chili flakes?)\b/],
+ ['canned-soup', /\b(chicken noodle soup|condensed soup|soup)\b/, /\b(soup mix|soup base|soup bowl)\b/],
+ ['ramen-12pk', /\b(ramen|instant noodles?)\b/],
+ ['mac-cheese', /\b(mac(aroni)? (and|&) cheese|macaroni cheese)\b/],
+ ['spaghetti', /\b(spaghetti|thin spaghetti)\b/],
+ ['white-rice', /\b(long grain white rice|white rice)\b/],
+ ['oatmeal', /\b(oatmeal|old fashioned oats|rolled oats|quick oats)\b/, /\b(oat milk|oat cereal|granola)\b/],
+ ['bread-white', /\b(white bread|white sandwich bread)\b/],
+ ['tortillas', /\btortillas?\b/],
+ ['jiffy-mix', /\b(corn muffin mix|muffin mix)\b/],
+ ['granola-bars', /\bgranola bars?\b/],
+ ['instant-coffee', /\binstant coffee\b/],
+ ['canned-beans', /\b(canned|black|kidney|pinto|white) beans\b/, /\b(dried|dry|green beans?|string beans?|bean sprouts?)\b/],
+ ['dried-beans', /\b(dried|dry) (pinto|black|kidney|white|great northern) beans\b/],
+ ['canned-corn', /\bcorn\b/, /\b(corn flakes?|corn chips?|corn starch|corn syrup|popcorn|cornmeal|corn dogs?)\b/],
+ ['baby-carrots', /\bbaby carrots?\b/],
+ ['canned-green-beans', /\bgreen beans?\b/],
+ ['frozen-mixed-veg', /\b(mixed vegetables?|frozen mixed veg)\b/],
+ ['onion-3lb', /\bonions?\b/, /\b(onion powder|onion rings?|onion soup)\b/],
+ ['canned-tomatoes', /\b(diced|crushed|whole) tomatoes?\b/],
+ ['potatoes-5lb', /\bpotatoes?\b/, /\b(chips?|crisps?|fries|flakes?|starch|salad)\b/],
+ ['bananas', /\bbananas?\b/, /\b(banana chips?|banana bread)\b/],
+ ['apples-3lb', /\bapples?\b/, /\b(apple juice|apple sauce|applesauce|apple cider|apple vinegar|pineapple)\b/],
+ ['oranges-3lb', /\boranges?\b/, /\b(orange juice|orange soda|orange chicken)\b/],
+ ['canned-fruit', /\b(peaches|fruit cocktail|canned fruit)\b/],
+ ['raisins', /\braisins?\b/],
+ ['strawberries', /\bstrawberries\b/],
+ ['cooking-oil', /\b(vegetable oil|canola oil|sunflower oil)\b/],
+ ['avocado', /\bavocados?\b/, /\b(oil|guacamole)\b/],
+ ['trail-mix', /\b(trail mix|mixed nuts)\b/],
+];
+
+/** A product's own name → the slug on the food list it labels, or null when the list has no such food. */
+function matchSlug(name) {
+ const s = String(name == null ? '' : name).normalize('NFKD').replace(/[\u0300-\u036f]/g, '')
+ .toLowerCase().replace(/&/g, ' and ').replace(/[^a-z0-9]+/g, ' ').trim();
+ if (!s) return null;
+ for (const [slug, yes, no] of MATCHERS) {
+ if (yes.test(s) && !(no && no.test(s))) return slug;
+ }
+ return null;
+}
+
+// food_off_products columns -> the Sources field they hold (the header has a '-' in two of them).
+const NUTRIENTS = {
+ energy_kcal_100g: 'energy_kcal_100g',
+ fat_100g: 'fat_100g',
+ saturated_fat_g: 'saturated_fat_100g',
+ carbs_g: 'carbohydrates_100g',
+ sugars_g: 'sugars_100g',
+ protein_g: 'proteins_100g',
+ salt_g: 'salt_100g',
+ fiber_g: 'fiber_100g',
+};
+
+/** A Sources item → the food_off_products row it states, or null when it is not a product this list has. */
+function rowOf(item) {
+ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'food_product') return null;
+ const code = String(item.identity == null ? '' : item.identity).trim();
+ if (!/^\d{6,14}$/.test(code)) return null;
+ const name = text(item.title, 300);
+ if (!name) return null;
+ const slug = matchSlug(name);
+ if (!slug) return null;
+ const f = item.fields || {};
+ const grade = text(f.nutriscore_grade, 3);
+ const row = {
+ code,
+ item_id: String(item.id || ''),
+ revision: Number(item.revision) || 1,
+ slug,
+ product_name: name,
+ brands: text(f.brands, 200),
+ product_url: item.canonical_url || `https://world.openfoodfacts.org/product/${code}`,
+ nutriscore_grade: grade && /^[a-e]$/i.test(grade) ? grade.toUpperCase() : null,
+ retrieved_at: (item.provenance && item.provenance.retrieved_at) || new Date(0).toISOString(),
+ };
+ for (const [col, field] of Object.entries(NUTRIENTS)) row[col] = num(f[field]);
+ if (Object.keys(NUTRIENTS).every((col) => row[col] == null) && !row.nutriscore_grade) return null;
+ return row;
+}
+
+/** The shape a food's page shows: the product as its label states it, per 100 g. */
+function shape(row) {
+ return {
+ code: row.code,
+ name: row.product_name,
+ brands: row.brands || null,
+ url: row.product_url || `https://world.openfoodfacts.org/product/${row.code}`,
+ nutriscore_grade: row.nutriscore_grade || null,
+ per_100g: {
+ energy_kcal: row.energy_kcal_100g == null ? null : Number(row.energy_kcal_100g),
+ fat_g: row.fat_100g == null ? null : Number(row.fat_100g),
+ saturated_fat_g: row.saturated_fat_g == null ? null : Number(row.saturated_fat_g),
+ carbs_g: row.carbs_g == null ? null : Number(row.carbs_g),
+ sugars_g: row.sugars_g == null ? null : Number(row.sugars_g),
+ protein_g: row.protein_g == null ? null : Number(row.protein_g),
+ salt_g: row.salt_g == null ? null : Number(row.salt_g),
+ fiber_g: row.fiber_g == null ? null : Number(row.fiber_g),
+ },
+ retrieved_at: row.retrieved_at,
+ note: NOTE,
+ license_url: ODBL_URL,
+ };
+}
+
+function createOff({ config, s, fetchImpl = globalThis.fetch, log = console, source = null }) {
+ const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');
+ const client = source || (url && config.oauth.clientSecret
+ ? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'food' } }, fetchImpl })
+ : null);
+ const cursor = ingest.createChangeCursor(s.db, { prefix: 'food', now: s.now });
+ const stats = { pulls: 0, applied: 0, removed: 0, last_at: null, last_error: null };
+ let byFood = new Map(); // food slug → shape(row), the most complete product for it
+ let timer = null;
+ let first = null;
+ let running = false;
+
+ async function apply(item, t) {
+ if (!item || item.source_key !== SOURCE_KEY || item.kind !== 'food_product') return 'hold';
+ if (item.removed) {
+ await t.query('DELETE FROM food_off_products WHERE item_id = $1', [String(item.id)]);
+ return 'removed';
+ }
+ const row = rowOf(item);
+ if (!row) return 'hold';
+ const cols = Object.keys(row);
+ await t.query(`INSERT INTO food_off_products (${cols.join(', ')}) VALUES (${cols.map((_, i) => `$${i + 1}`).join(', ')})
+ ON CONFLICT (code) DO UPDATE SET ${cols.filter((c) => c !== 'code').map((c) => `${c} = EXCLUDED.${c}`).join(', ')}
+ WHERE food_off_products.item_id <> EXCLUDED.item_id OR food_off_products.revision <= EXCLUDED.revision`, cols.map((c) => row[c]));
+ return 'applied';
+ }
+
+ /** One product per food: the one that states the most nutrient values (then its Nutri-Score). */
+ async function load() {
+ const rows = await s.db.many('SELECT * FROM food_off_products');
+ const best = new Map();
+ const score = (r) => Object.keys(NUTRIENTS).filter((col) => r[col] != null).length + (r.nutriscore_grade ? 1 : 0);
+ for (const row of rows) {
+ const cur = best.get(row.slug);
+ if (!cur || score(row) > score(cur) || (score(row) === score(cur) && row.code < cur.code)) best.set(row.slug, row);
+ }
+ byFood = new Map([...best.entries()].map(([slug, row]) => [slug, shape(row)]));
+ return byFood.size;
+ }
+
+ /** Follow Sources' change feed to its end. Never throws: the caller is a timer or a test. */
+ async function pull() {
+ if (!client || running) return { skipped: true };
+ running = true;
+ try {
+ const out = await ingest.pullChanges({ db: s.db, cursor, source: client, name: CURSOR, apply });
+ stats.pulls++;
+ stats.applied += out.applied;
+ stats.removed += out.removed;
+ stats.last_at = s.iso();
+ stats.last_error = null;
+ await load();
+ if (out.applied || out.removed) log.log(`[OFF] ${out.applied} Open Food Facts products updated, ${out.removed} removed (Sources cursor ${out.after})`);
+ return out;
+ } catch (err) {
+ stats.last_error = text(err && err.message, 300);
+ log.warn(`[OFF] pull from OpenVibe.Sources failed: ${stats.last_error}`);
+ return { error: stats.last_error };
+ } finally { running = false; }
+ }
+
+ return {
+ enabled: Boolean(client),
+ stats,
+ pull,
+ apply,
+ load,
+ /** The Open Food Facts product for a food on the list, or null when no product matched (or nothing is read yet). */
+ forFood(slug) { return byFood.get(slug) || null; },
+ start() {
+ load().catch(() => { /* the first pull loads it */ });
+ if (!client || timer) return;
+ first = setTimeout(() => { pull(); }, FIRST_MS);
+ timer = setInterval(() => { pull(); }, EVERY_MS);
+ if (first.unref) first.unref();
+ if (timer.unref) timer.unref();
+ },
+ stop() {
+ if (first) clearTimeout(first);
+ if (timer) clearInterval(timer);
+ first = null;
+ timer = null;
+ },
+ };
+}
+
+module.exports = { createOff, rowOf, shape, matchSlug, MATCHERS, SOURCE_KEY, NOTE, ODBL_URL };
@@ -597,6 +597,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'
@@ -597,6 +597,7 @@ ${list.length ? sections : notice('Nothing matched. Try a shorter word.', 'warn'
r.get('/foods/:slug', (req, res) => {
r.get('/foods/:slug', (req, res) => {
const f = foods.getFood(req.params.slug);
const f = foods.getFood(req.params.slug);
const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;
const official = f && ctx.officialPrices ? ctx.officialPrices.forFood(f.slug) : null;
+ const off = f && ctx.off ? ctx.off.forFood(f.slug) : null;
if (!f) return page(req, res, {
if (!f) return page(req, res, {
title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],
title: 'No such food', crumbs: [{ label: 'Food list', href: '/foods' }, { label: 'Not found' }],
body: html`<h1>No such food</h1><p>Nothing in the list is called “${String(req.params.slug).slice(0, 64)}”. <a href="/foods">See the whole list</a>.</p>`,
body: html`<h1>No such food</h1><p>Nothing in the list is called “${String(req.params.slug).slice(0, 64)}”. <a href="/foods">See the whole list</a>.</p>`,
@@ -623,6 +624,19 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])
@@ -623,6 +624,19 @@ ${table(['Store', 'A pack'], f.prices.map((p) => [p.store_name, money(p.price)])
${official ? html`<h2>US average price</h2>
${official ? html`<h2>US average price</h2>
<p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p>
<p><b>${money(official.pack_price)}</b> for this pack at the US city average for ${official.month}: ${money(official.value)} per ${official.unit} of ${official.item.charAt(0).toLowerCase() + official.item.slice(1)} (<a href="${official.url}" rel="noopener">BLS series ${official.series_id}</a>).</p>
<p class="muted small">${OFFICIAL_NOTE}</p>` : ''}
<p class="muted small">${OFFICIAL_NOTE}</p>` : ''}
+${off ? html`<h2>Open Food Facts label</h2>
+<p><b>${off.name}</b>${off.brands ? html` · ${off.brands}` : ''}${off.nutriscore_grade ? html` · Nutri-Score ${off.nutriscore_grade}` : ''}</p>
+${table(['Per 100 g', 'As the label states'], [
+ ['Energy', off.per_100g.energy_kcal == null ? '—' : `${off.per_100g.energy_kcal} kcal`],
+ ['Fat', off.per_100g.fat_g == null ? '—' : `${off.per_100g.fat_g} g`],
+ ['of which saturates', off.per_100g.saturated_fat_g == null ? '—' : `${off.per_100g.saturated_fat_g} g`],
+ ['Carbohydrate', off.per_100g.carbs_g == null ? '—' : `${off.per_100g.carbs_g} g`],
+ ['of which sugars', off.per_100g.sugars_g == null ? '—' : `${off.per_100g.sugars_g} g`],
+ ['Protein', off.per_100g.protein_g == null ? '—' : `${off.per_100g.protein_g} g`],
+ ['Salt', off.per_100g.salt_g == null ? '—' : `${off.per_100g.salt_g} g`],
+ ['Fibre', off.per_100g.fiber_g == null ? '—' : `${off.per_100g.fiber_g} g`],
+ ])}
+<p class="muted small"><a href="${off.url}" rel="noopener">Open Food Facts (ODbL)</a> · product ${off.code}${off.retrieved_at ? html`, read ${time(off.retrieved_at)}` : ''}. ${off.note} These are the product's own label figures, shown beside the typical values above and not replacing them.</p>` : ''}
${f.tags.length ? html`<p class="tags">${f.tags.map((t) => badge(t))}</p>` : ''}
${f.tags.length ? html`<p class="tags">${f.tags.map((t) => badge(t))}</p>` : ''}
<p><a href="/plan?people=2&days=3">Plan meals with this</a> · <a href="/foods">Back to the list</a></p>`,
<p><a href="/plan?people=2&days=3">Plan meals with this</a> · <a href="/foods">Back to the list</a></p>`,
});
});
@@ -44,13 +44,14 @@ async function start() {
@@ -44,13 +44,14 @@ async function start() {
// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.
// The recipe library writes a theme every six hours (server/food/library.js); off without OpenVibe.AI.
ctx.library.start();
ctx.library.start();
ctx.recipeSearch.start();
ctx.recipeSearch.start();
- // Food recalls and BLS's average prices follow OpenVibe.Sources' change feed hourly (server/food/recalls.js,
- // server/food/official-prices.js).
+ // Food recalls, BLS's average prices and Open Food Facts products follow OpenVibe.Sources' change feed
+ // (server/food/recalls.js, server/food/official-prices.js, server/food/off.js).
ctx.recalls.start();
ctx.recalls.start();
ctx.officialPrices.start();
ctx.officialPrices.start();
+ ctx.off.start();
// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).
// The home's "food near" cities: their area searches refreshed in the background (server/food/near-warm.js).
ctx.nearWarm.start();
ctx.nearWarm.start();
- const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.nearWarm.stop()];
+ const extra = [() => { if (subscriptions) subscriptions.stop(); }, () => ctx.library.stop(), () => ctx.recipeSearch.stop(), () => ctx.recalls.stop(), () => ctx.officialPrices.stop(), () => ctx.off.stop(), () => ctx.nearWarm.stop()];
createLifecycle({ server, ctx, extra });
createLifecycle({ server, ctx, extra });
return { server, ctx };
return { server, ctx };
}
}
@@ -50,7 +50,7 @@ async function boot(opts = {}) {
@@ -50,7 +50,7 @@ async function boot(opts = {}) {
const { createStore } = require('../../server/db');
const { createStore } = require('../../server/db');
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
const store = createStore(testdb.db, { now: opts.now });
const store = createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource });
+ const built = await createApp({ config, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, accountSend: opts.accountSend, ai: opts.ai, barcodes: opts.barcodes, recallSource: opts.recallSource, priceSource: opts.priceSource, offSource: opts.offSource });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
new file mode 100644
new file mode 100644
@@ -0,0 +1,109 @@
@@ -0,0 +1,109 @@
+'use strict';
+/**
+ * Open Food Facts (server/food/off.js): products from OpenVibe.Sources as label data beside the food list.
+ * - a product's own name maps to one of the list's foods; chips, juice and flakes do not become the food;
+ * - the pull follows the change feed from its own cursor and keeps the most complete product for a food;
+ * - a food's page shows the product's per-100 g figures with the Open Food Facts (ODbL) credit and link, beside
+ * the list's own nutrition and prices, never instead of them;
+ * - the match survives a restart because it is in food_off_products.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { rowOf, matchSlug, shape, NOTE } = require('../server/food/off');
+const { createOff } = require('../server/food/off');
+
+const item = (code, title, fields = {}, over = {}) => ({
+ id: `itm_off_${code}`, source_key: 'open-food-facts', category: 'food', kind: 'food_product', identity: code,
+ canonical_url: `https://world.openfoodfacts.org/product/${code}`, title, summary: null,
+ fields: { brands: 'Example Brand', nutriscore_grade: 'b', ...fields },
+ revision: 1, provenance: { retrieved_at: '2026-10-30T06:00:00.000Z' }, removed: null, ...over,
+});
+
+function feed(items) {
+ return {
+ items,
+ async listItems({ after = 0, limit = 100 }) {
+ const rest = items.filter((x) => x.change_seq > after).sort((a, b) => a.change_seq - b.change_seq);
+ const page = rest.slice(0, limit);
+ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: rest.length > limit, sources: {} };
+ },
+ };
+}
+
+(async () => {
+ const items = [
+ { ...item('0000000000011', 'Large Eggs', { energy_kcal_100g: '143', proteins_100g: '12.6' }), change_seq: 1 },
+ { ...item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '140', proteins_100g: '13', fat_100g: '9.5', carbohydrates_100g: '0.7', salt_100g: '0.3' }, { revision: 3 }), change_seq: 2 },
+ { ...item('0000000000021', 'Whole Milk', { energy_kcal_100g: '64', proteins_100g: '3.3', fat_100g: '3.6', carbohydrates_100g: '4.8', sugars_100g: '4.8' }), change_seq: 3 },
+ { ...item('0000000000031', 'Banana Chips', { energy_kcal_100g: '519' }), change_seq: 4 }, // a snack, not a banana
+ { ...item('0000000000041', 'Potato Chips', { energy_kcal_100g: '536' }), change_seq: 5 }, // not the potato on the list
+ { ...item('0000000000051', 'Apple Juice', { energy_kcal_100g: '46' }), change_seq: 6 }, // not an apple
+ { ...item('0000000000061', 'Mystery Product', {}), change_seq: 7 }, // no figures, no match
+ { ...item('0000000000071', 'Whole Milk', { proteins_100g: '3.3' }, { kind: 'record' }), change_seq: 8 },
+ { ...item('0000000000081', 'Whole Milk', { proteins_100g: '3.3' }, { source_key: 'openfda-food-enforcement' }), change_seq: 9 },
+ ];
+ const source = feed(items);
+ const t = await boot({ offSource: source });
+ try {
+ await check('names map to the list\'s foods, and a snack, a juice or a chip does not', async () => {
+ assert.strictEqual(matchSlug('Peanut Butter, Creamy'), 'peanut-butter', 'peanut butter is not butter');
+ assert.strictEqual(matchSlug('Whole Milk'), 'whole-milk');
+ assert.strictEqual(matchSlug('Cheddar Cheese Block'), 'cheese-block');
+ assert.strictEqual(matchSlug('Dried Pinto Beans'), 'dried-beans');
+ assert.strictEqual(matchSlug('Canned Black Beans'), 'canned-beans');
+ assert.strictEqual(matchSlug('Banana Chips'), null);
+ assert.strictEqual(matchSlug('Potato Chips'), null);
+ assert.strictEqual(matchSlug('Apple Juice'), null);
+ assert.strictEqual(matchSlug('Corn Flakes'), null);
+ const row = rowOf(item('0000000000021', 'Whole Milk', { energy_kcal_100g: '64', proteins_100g: '3.3' }));
+ assert.deepStrictEqual([row.slug, row.product_name, row.energy_kcal_100g, row.protein_g], ['whole-milk', 'Whole Milk', 64, 3.3]);
+ assert.strictEqual(rowOf(item('0000000000061', 'Mystery Product', {})), null, 'no nutrient and no Nutri-Score = nothing to show');
+ assert.strictEqual(rowOf(item('abc', 'Whole Milk')), null, 'a code that is not a product code');
+ });
+
+ await check('the pull follows the feed: matches applied, everything else held', async () => {
+ const out = await t.ctx.off.pull();
+ assert.deepStrictEqual([out.applied, out.hold], [3, 6], 'eggs twice, milk once; the snacks and the two wrong items held');
+ assert.strictEqual(Number(await t.ctx.s.db.value("SELECT cursor FROM food_ingest_cursor WHERE name = 'sources.food.off.after'")), 9);
+ const eggs = t.ctx.off.forFood('eggs-dozen');
+ assert.strictEqual(eggs.code, '0000000000012', 'the more complete product is the one shown');
+ assert.deepStrictEqual([eggs.name, eggs.brands, eggs.nutriscore_grade], ['Free Range Large Eggs', 'Example Brand', 'B']);
+ assert.deepStrictEqual([eggs.per_100g.energy_kcal, eggs.per_100g.protein_g, eggs.per_100g.salt_g], [140, 13, 0.3]);
+ assert.strictEqual(t.ctx.off.forFood('canned-tuna'), null, 'a food no product matched shows none');
+ });
+
+ await check('a food\'s page shows the product with the ODbL credit, beside its own nutrition', async () => {
+ const page = await t.get('/foods/eggs-dozen');
+ assert.strictEqual(page.status, 200);
+ assert.ok(page.text.includes('Open Food Facts label'));
+ assert.ok(page.text.includes('Open Food Facts (ODbL)'), 'the exact credit');
+ assert.ok(page.text.includes('https://world.openfoodfacts.org/product/0000000000012'), 'the product is linked');
+ assert.ok(page.text.includes('Free Range Large Eggs'));
+ assert.ok(page.text.includes('140 kcal') && page.text.includes('13 g'), 'the label figures');
+ assert.ok(page.text.includes('Nutrition') && page.text.includes('A serving'), 'the list\'s own section is still there');
+ assert.ok(page.text.includes(NOTE.split('.')[0]), 'the ODbL note');
+ const tuna = await t.get('/foods/canned-tuna');
+ assert.ok(!tuna.text.includes('Open Food Facts label'), 'a food with no match shows no label section');
+ });
+
+ await check('the match is durable: a new instance loads it from food_off_products', async () => {
+ const again = createOff({ config: { sources: { internalUrl: '' } }, s: t.ctx.s, log: { log() {}, warn() {} } });
+ assert.strictEqual(await again.load(), 2);
+ assert.strictEqual(again.forFood('eggs-dozen').code, '0000000000012');
+ const shapeOf = shape({ ...rowOf(item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '140' })), retrieved_at: '2026-10-30T06:00:00.000Z' });
+ assert.strictEqual(shapeOf.license_url, 'https://opendatacommons.org/licenses/odbl/1-0/');
+ });
+
+ await check('a removed product is deleted; an older revision never wins', async () => {
+ source.items.push({ ...item('0000000000021', 'Whole Milk', {}, { id: 'itm_off_0000000000021', removed: { at: '2026-10-30T07:00:00.000Z', reason: 'licence' }, revision: 2 }), change_seq: 10 });
+ source.items.push({ ...item('0000000000012', 'Free Range Large Eggs', { energy_kcal_100g: '1' }, { id: 'itm_off_0000000000012', revision: 1 }), change_seq: 11 });
+ const out = await t.ctx.off.pull();
+ assert.strictEqual(out.removed, 1);
+ assert.strictEqual(t.ctx.off.forFood('whole-milk'), null);
+ assert.strictEqual(t.ctx.off.forFood('eggs-dozen').per_100g.energy_kcal, 140, 'the older revision did not overwrite the label');
+ });
+ } finally {
+ await t.close();
+ }
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });