openvibers/openvibe.sources Commits

USDA ERS fruit and vegetable prices: CSV api format, composite identity, disabled seed (#205)

OP OpenVibers committed 7 days ago · 9582aff7e2 · 10 files, +218 −15

* An api endpoint may read a comma-separated file and join several columns into its identity; USDA ERS fruit and vegetable prices seeded (disabled)

ERS publishes its annual fruit and vegetable prices as two CSV tables (all fruits, all vegetables): a UTF-8 BOM, quoted names with commas, prices in dollars per pound or pint. The api adapter gains format csv (RFC 4180 quoting) and identity_fields, because no single column is unique (Apples has four forms). The seed keeps every row of both published tables, one item per product and form, disabled for terms review; Food maps 16 of them. Robots checked 2026-11-05 from production; terms quoted in the seed. Needs openvibe-contracts: format csv and identity_fields (the local proposal is updated here).

* Merge remote-tracking branch 'origin/main' into usda-ers-sources

* api@1: identity_fields paths use pathValid after the #206 selector rewrite

The merge kept identity_fields' path check on PATH_RE, which #206 removed when selector values became nested paths; a CSV seed with identity_fields would have thrown a ReferenceError at registry validation. The check now uses pathValid, as every other endpoint path does.

* openvibe-contracts v0.290.0 (was v0.289.1)

Pinned after the v0.290.0 release: sources.source@1 now carries api format csv and identity_fields, so the seed's additions are released contract rather than only the local proposal.

Changes

ddiff --git a/README.md b/README.md
iindex 89348d6..552b356 100644
---- a/README.md
++++ b/README.md
@@@ -105,7 +105,7 @@ succeeded. Health status: `healthy | stale | failing | never_fetched | disabled
  | `rss`, `atom` (`feed@3`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars; a deal feed's own offer elements (DealNews' `dealnews:price` with its currency, `dealnews:retailer`, `dealnews:expires`) become `fields.price`, `currency`, `seller` and `valid_until` exactly as stated (the item stays an article; nothing is read out of the headline); its `dealnews:category` fills `fields.categories` when the item has no `<category>`, and `dealnews:staffPick` is `fields.staff_pick` (true/false, absent when not stated) |
  | `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |
  | `jsonld` (`jsonld@2`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error`; `list_price` from a ListPrice/StrikethroughPrice/MSRP/SRP `priceSpecification`, as stated |
--| `api` (`api@1`) | JSON or XML official APIs via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead |
++| `api` (`api@1`) | JSON, XML, TSV or CSV official files via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead |
  | `bulk` (`bulk@1`) | one dataset file, streamed (CSV, JSONL or TSV) | `max_bytes` is the size cap, `max_rows` the per-run row limit; the stream stops at the limit, so a multi-GB file is never held in memory. `resolver: "off-delta"` reads a text index and only the gzipped JSONL delta parts it lists, keeping the last finished part and the current part's row offset in `source_state` (Open Food Facts) |
  | `crawl` (`jsonld@2`, `page@1`) | a whole site from its seeds: its sitemaps and, if asked, its links (see below) | only pages that publish JSON-LD are kept, unless `require_structured: false` keeps a `page` item from the page's own search metadata |
  | `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |
@@@ -122,6 +122,13 @@ an empty success. With the default `items_shape: "array"`, `$index` is a record'
  the array (as a whole path, e.g. `"extra": { "position": "$index" }`), so a ranked list keeps its
  order (Steam's top sellers).
  
++A CSV or TSV body likewise names its columns on the first line: each further line is a record of
++trimmed text, and `items_path` must be "". CSV follows RFC 4180 quoting (a quoted field may
++hold commas, doubled quotes and line breaks); a row whose value count differs from the header is
++skipped, and a body that ends inside a quoted field is a parse error. When no single column is unique,
++`identity_fields` names two to eight paths whose values, joined with " | ", are the record's identity
++(the USDA ERS table prices Apples in four forms); `fields.identity` cannot be given with it.
++
  ```json
  { "url": "https://api.porkbun.com/api/json/v3/pricing/get", "format": "json",
    "items_path": "pricing", "items_shape": "map", "item_kind": "tld",
@@@ -591,7 +598,7 @@ in [OpenVibers/OpenVibe.Sites](https://github.com/OpenVibers/OpenVibe.Sites); th
  Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
  
  <!-- versions:start -->
--- openvibe-contracts: v0.289.1
++- openvibe-contracts: v0.290.0
  - openvibe-sdk: v0.82.0
  - openvibe-shared: v4.109.2
  <!-- versions:end -->
ddiff --git a/STATUS.json b/STATUS.json
iindex 5532fd8..ad4ebd2 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -7,7 +7,7 @@
    "runtime": "node 22, express, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9, fast-xml-parser",
    "port": 4720,
    "domain": "sources.openvibe.network (vhost in deploy/nginx: an internal-service page, the merchant opt-in pages, health and JSON 404s; the JSON API is loopback-only; README \"Public host\")",
--  "contracts": "openvibe-contracts v0.289.1",
++  "contracts": "openvibe-contracts v0.290.0",
    "sdk": "openvibe-sdk v0.82.0",
    "shared": "openvibe-shared v4.109.2",
    "capabilities": [
@@@ -51,5 +51,5 @@
      "staff-only index documents for Search",
      "per-actor limits (openvibe-sdk/limits)"
    ],
--  "updated": "2026-11-05"
++  "updated": "2026-11-06"
  }
ddiff --git a/docs/contracts-proposal/contracts/sources/source.v1.json b/docs/contracts-proposal/contracts/sources/source.v1.json
iindex 8ca161a..242fbea 100644
---- a/docs/contracts-proposal/contracts/sources/source.v1.json
++++ b/docs/contracts-proposal/contracts/sources/source.v1.json
@@@ -89,7 +89,16 @@
                "csv",
                "jsonl"
              ],
--            "description": "json (default), xml, tsv (a tab-separated file whose first line names the columns; BLS series files, Sources 2026-10-24), or, for a bulk source, csv, jsonl or tsv: one dataset file read as a stream, row by row (Sources, 2026-10-31)."
++            "description": "json (default), xml, tsv (a tab-separated file whose first line names the columns; BLS series files, Sources 2026-10-24), csv (a comma-separated file with RFC 4180 quoting; USDA ERS fruit and vegetable prices, Sources 2026-11-05), or, for a bulk source, csv, jsonl or tsv: one dataset file read as a stream, row by row (Sources, 2026-10-31)."
++          },
++          "identity_fields": {
++            "type": "array",
++            "minItems": 2,
++            "maxItems": 8,
++            "items": {
++              "type": "string"
++            },
++            "description": "A record's identity from several columns, joined in order with \" | \", for a table where no single column is unique (USDA ERS prices a product in several forms; Sources 2026-11-05). Requires items_shape array and cannot be given with fields.identity."
            },
            "duplicates": {
              "enum": [
ddiff --git a/package-lock.json b/package-lock.json
iindex d794d37..fe0b068 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -12,7 +12,7 @@
          "dotenv": "^16.4.7",
          "express": "^4.21.2",
          "fast-xml-parser": "^4.5.3",
--        "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.289.1",
++        "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.290.0",
          "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
          "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",
          "pg": "^8.23.0"
@@@ -1044,9 +1044,9 @@
        }
      },
      "node_modules/openvibe-contracts": {
--      "version": "0.289.1",
--      "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.289.1",
--      "integrity": "sha512-LrKTwe8VPCWn+b0MTNmfjAYEJ0rtgLxeM6EHONBDl0ynt/Igndz+ruDEpwuz5SZ9HYCIMkCqSV5ujhedh6zmzQ==",
++      "version": "0.290.0",
++      "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.290.0",
++      "integrity": "sha512-WX9p49Wyr5dPRQ9iQEJvVEPIQwkJYsJTLryU5DJJI/GrxzfMZFN1nfuM5W/KmL3DGoRTAOg/KAWjs0OkunJyeg==",
        "license": "AGPL-3.0-only",
        "dependencies": {
          "ajv": "^8.17.1",
ddiff --git a/package.json b/package.json
iindex ec5b26e..704454e 100644
---- a/package.json
++++ b/package.json
@@@ -30,7 +30,7 @@
      "dotenv": "^16.4.7",
      "express": "^4.21.2",
      "fast-xml-parser": "^4.5.3",
--    "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.289.1",
++    "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.290.0",
      "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.82.0",
      "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.109.2",
      "pg": "^8.23.0"
ddiff --git a/seeds/sources.json b/seeds/sources.json
iindex a0ef4b7..30a40e5 100644
---- a/seeds/sources.json
++++ b/seeds/sources.json
@@@ -1972,6 +1972,53 @@
        "sensitivity": "none",
        "default_indexability": "noindex"
      },
++    {
++      "key": "usda-ers-fruit-vegetable-prices",
++      "name": "USDA ERS average retail prices for fruits and vegetables",
++      "type": "api",
++      "category": "food",
++      "homepage_url": "https://www.ers.usda.gov/data-products/fruit-and-vegetable-prices/",
++      "endpoints": [
++        {
++          "url": "https://www.ers.usda.gov/media/6210/all-fruits-average-prices-csv-format.csv",
++          "format": "csv",
++          "items_path": "",
++          "item_kind": "price",
++          "identity_fields": ["Fruit", "Form"],
++          "fields": { "title": "Fruit" },
++          "extra": {
++            "name": "Fruit",
++            "form": "Form",
++            "price": "AverageRetailPrice",
++            "unit": "AverageRetailPriceUnitOfMeasure"
++          }
++        },
++        {
++          "url": "https://www.ers.usda.gov/media/6240/all-vegetables-average-prices-csv-format.csv",
++          "format": "csv",
++          "items_path": "",
++          "item_kind": "price",
++          "identity_fields": ["Vegetable", "Form"],
++          "fields": { "title": "Vegetable" },
++          "extra": {
++            "name": "Vegetable",
++            "form": "Form",
++            "price": "AverageRetailPrice",
++            "unit": "AverageRetailPriceUnitOfMeasure"
++          }
++        }
++      ],
++      "robots_note": "robots.txt checked 2026-11-05 from the production host: User-agent * Allow /, with only /core/, /profiles/, /README.md and /web.config disallowed; the two /media/ CSV paths are allowed. The fetcher re-checks it on every run. Two CSV files, about 5 KB and 7 KB, one request each a run.",
++      "terms_note": "Verified 2026-11-05 from the production host: USDA's Economic Research Service is a US federal agency, and the page says its fruit and vegetable prices are ERS's own estimates from Circana scanner data ('ERS calculated average prices at retail stores using 2013, 2016, 2020, 2022, and 2023 retail scanner data from Circana'). The documentation asks for the citation 'U.S. Department of Agriculture, Economic Research Service. (2025). Fruit and Vegetable Prices.' and the page says 'The findings in this data product or stated on this page should not be attributed to Circana.' The prices are annual national averages, updated 'each year, subject to data availability' (files last updated 2025-12-09), and the page says twice that the data 'should not be used for making inferences about price changes over time'. Kept: every row of the two published all-fruits and all-vegetables tables (product, form, average retail price per pound or pint, unit), one item per product and form. Food shows it as an annual national average with its 2023 data year, cites ERS and repeats the no-trend rule; nothing is compared over time. Re-verify before enabling.",
++      "license_note": "US federal government work (Economic Research Service, U.S. Department of Agriculture), public domain in the US; ERS is cited as its documentation asks, and no finding is attributed to Circana.",
++      "min_interval_ms": 60000,
++      "poll_interval_sec": 604800,
++      "max_items": 250,
++      "enabled": false,
++      "review_required": true,
++      "sensitivity": "none",
++      "default_indexability": "noindex"
++    },
  {
        "key": "sec-press",
        "name": "US Securities and Exchange Commission",
ddiff --git a/server/adapters/api.js b/server/adapters/api.js
iindex 32ba95b..5bb28cf 100644
---- a/server/adapters/api.js
++++ b/server/adapters/api.js
@@@ -79,6 +79,15 @@
   * value). A line with a different number of values than the header is skipped; a file without a header is a parse
   * error.
   *
++ * `"format": "csv"` reads a comma-separated file the same way, with RFC 4180 quoting (a quoted field may hold
++ * commas, doubled quotes and line breaks); the header line names the columns. The USDA Economic Research Service
++ * publishes its fruit and vegetable prices this way (ers.usda.gov). A row with a different number of values than the
++ * header, or a body that ends inside a quoted field, is a parse error or a skipped row, never invented text.
++ *
++ * An endpoint may name its identity as several columns instead of one: `"identity_fields": ["Fruit", "Form"]`
++ * joins their values in order with " | " (a row missing any of them is skipped). This is what a table needs when
++ * no single column is unique — the USDA table has Apples in four forms. `fields.identity` cannot be given with it.
++ *
   * Two records with the same identity keep the first (`"duplicates": "first"`, the default). `"duplicates": "last"`
   * keeps the last instead, in the first one's place: a file that lists each series oldest to newest then yields its
   * newest value.
@@@ -322,15 +331,26 @@ function checkConditions(where, label, shape) {
  
  /** Validate an endpoint mapping (registry side). `bulk` is the streaming dataset download's own schema. */
  function checkMapping(ep, { bulk = false } = {}) {
--    const formats = bulk ? ['csv', 'jsonl', 'tsv'] : ['json', 'xml', 'tsv'];
--    if (!formats.includes(ep.format || (bulk ? 'csv' : 'json'))) return bulk ? 'format must be csv, jsonl or tsv' : 'format must be json, xml or tsv';
++    const formats = bulk ? ['csv', 'jsonl', 'tsv'] : ['json', 'xml', 'tsv', 'csv'];
++    if (!formats.includes(ep.format || (bulk ? 'csv' : 'json'))) return bulk ? 'format must be csv, jsonl or tsv' : 'format must be json, xml, tsv or csv';
      if (['tsv', 'csv', 'jsonl'].includes(ep.format) && ep.items_path) return `a ${ep.format} body is the array itself: items_path is ""`;
      if (!bulk && ep.duplicates !== undefined && !['first', 'last'].includes(ep.duplicates)) return 'duplicates must be first or last';
      const shape = ep.items_shape || 'array';
      if (!bulk && !['array', 'map'].includes(shape)) return 'items_shape must be array or map';
      if (!bulk && ep.items_path !== undefined && ep.items_path !== '' && !pathValid(ep.items_path)) return 'items_path is malformed';
      const fields = ep.fields || {};
--    if (!fields.identity && !fields.url) return 'fields.identity or fields.url is required';
++    const identityFields = ep.identity_fields;
++    if (identityFields !== undefined) {
++        if (!Array.isArray(identityFields) || identityFields.length < 2 || identityFields.length > 8) return 'identity_fields must be an array of 2 to 8 paths';
++        if (shape !== 'array') return 'identity_fields requires items_shape "array"';
++        if (fields.identity) return 'identity_fields and fields.identity cannot both be given';
++        for (const p of identityFields) {
++            if (typeof p !== 'string' || !pathValid(p)) return 'identity_fields paths must be paths';
++            const kerr = keyPathViolation(p, shape);
++            if (kerr) return `identity_fields: ${kerr}`;
++        }
++    }
++    if (!fields.identity && !fields.url && !identityFields) return 'fields.identity, fields.url or identity_fields is required';
      for (const [k, p] of Object.entries(fields)) {
          if (!['identity', 'url', 'title', 'summary', 'author', 'published_at', 'published_at_epoch', 'updated_at', 'updated_at_epoch'].includes(k)) return `unknown field ${k}`;
          if (typeof p !== 'string' || !pathValid(p)) return `fields.${k} path is malformed`;
@@@ -387,12 +407,56 @@ function parseTsv(text) {
      return { records, bad };
  }
  
++/** A comma-separated body as records keyed by its header line's names (RFC 4180 quoting); malformed rows are counted. */
++function parseCsv(text) {
++    const src = String(text).replace(/^\uFEFF/, '');
++    const rows = [];
++    let row = [];
++    let field = '';
++    let quoted = false;
++    for (let i = 0; i < src.length; i++) {
++        const ch = src[i];
++        if (quoted) {
++            if (ch === '"') {
++                if (src[i + 1] === '"') { field += '"'; i++; }
++                else quoted = false;
++            } else field += ch;
++            continue;
++        }
++        if (ch === '"' && field === '') { quoted = true; continue; }
++        if (ch === ',') { row.push(field); field = ''; continue; }
++        if (ch === '\r') continue;
++        if (ch === '\n') { row.push(field); rows.push(row); row = []; field = ''; continue; }
++        field += ch;
++    }
++    if (quoted) throw new ParseError('csv body ends inside a quoted field');
++    if (field !== '' || row.length) { row.push(field); rows.push(row); }
++    const lines = rows.filter((r) => !(r.length === 1 && String(r[0]).trim() === ''));
++    if (!lines.length) throw new ParseError('csv body has no header line');
++    const names = lines[0].map((s) => String(s).trim());
++    if (names.some((n) => !n) || new Set(names).size !== names.length) throw new ParseError('csv header names must be present and distinct');
++    const records = [];
++    let bad = 0;
++    for (const values of lines.slice(1)) {
++        if (values.length !== names.length) { bad++; continue; }
++        const rec = {};
++        names.forEach((n, i) => { rec[n] = String(values[i]).trim(); });
++        records.push(rec);
++    }
++    return { records, bad };
++}
++
  /** One record → the item every api-shaped adapter stores, or null when it states no identity. */
  function mapRecord(body, rec, key, index, endpoint, url, summaryMax = 1000) {
      const f = endpoint.fields || {};
      const get = (p) => (p === '$index' ? index : getField(body, rec, key, p));
      const link = f.url ? canonicalUrl(scalar(get(f.url)), url) : null;
--    const identity = (f.identity ? toId(scalar(get(f.identity))) : null) || link;
++    let identity = null;
++    if (Array.isArray(endpoint.identity_fields)) {
++        const parts = endpoint.identity_fields.map((p) => scalar(get(p)));
++        if (parts.every((v) => v != null && v !== '')) identity = toId(parts.join(' | '));
++    } else if (f.identity) identity = toId(scalar(get(f.identity)));
++    identity = identity || link;
      if (!identity) return null;
      const extra = {};
      for (const [k, p] of Object.entries(endpoint.extra || {})) extra[k] = scalar(get(p));
@@@ -416,6 +480,7 @@ function parse(text, { url, endpoint = {}, summaryMax = 1000, maxItems = 500 } =
      let malformed = 0;
      if (format === 'xml') body = parseXml(text);
      else if (format === 'tsv') ({ records: body, bad: malformed } = parseTsv(text));
++    else if (format === 'csv') ({ records: body, bad: malformed } = parseCsv(text));
      else {
          try { body = JSON.parse(text); } catch { throw new ParseError('body is not valid JSON'); }
      }
ddiff --git a/server/registry.js b/server/registry.js
iindex 3495557..47f563d 100644
---- a/server/registry.js
++++ b/server/registry.js
@@@ -135,6 +135,7 @@ function validateSource(input, { maxItemsCap = 12000 } = {}) {
              if (e.filter) out.filter = e.filter;
              // and `extra_lists` (tags, job types) the same way.
              if (e.extra_lists) out.extra_lists = e.extra_lists;
++            if (e.identity_fields !== undefined) out.identity_fields = e.identity_fields;
              if (e.duplicates !== undefined) out.duplicates = e.duplicates;
              if (e.max_bytes !== undefined) out.max_bytes = e.max_bytes;
              if (e.timeout_ms !== undefined) out.timeout_ms = e.timeout_ms;
ddiff --git a/test/api-csv.test.js b/test/api-csv.test.js
nnew file mode 100644
iindex 0000000..d1f8a9a
---- /dev/null
++++ b/test/api-csv.test.js
@@@ -0,0 +1,74 @@
++'use strict';
++/**
++ * API adapter: `format: "csv"` reads a comma-separated file by its header, and `identity_fields` makes the identity
++ * from several columns when no single one is unique (USDA ERS prices each product in several forms).
++ */
++const assert = require('assert');
++const api = require('../server/adapters/api');
++const { suite } = require('./helpers');
++
++const t = suite('api-csv');
++
++// The USDA ERS all-fruits file as published: UTF-8 BOM, quoted names with commas, padding, and a column per product.
++const USDA = [
++    '\uFEFFFruit,Form,AverageRetailPrice,AverageRetailPriceUnitOfMeasure,AveragePricePerCupEquivalent',
++    'Apples,Fresh,1.86093095,per pound,0.501435057',
++    '"Apples, applesauce",Canned,1.208099427,per pound,0.652533815',
++    'Bananas,Fresh,0.609594742,per pound,0.31498274',
++    '"Grapes (raisins)",Dried,4.328343983,per pound,',
++    'Oranges,Fresh,1.439704016,per pound,0.863515806',
++    'a broken row with too few values',
++    '',
++    'Strawberries,Fresh,3.078734363,per pound,1.046999868',
++].join('\r\n');
++
++const usdaEndpoint = {
++    format: 'csv', items_path: '', item_kind: 'price',
++    identity_fields: ['Fruit', 'Form'],
++    fields: { title: 'Fruit' },
++    extra: { name: 'Fruit', form: 'Form', price: 'AverageRetailPrice', unit: 'AverageRetailPriceUnitOfMeasure' },
++};
++
++t('a USDA table: one item per product and form, joined as "name | form"; quoted commas kept, broken rows skipped', () => {
++    assert.strictEqual(api.checkMapping(usdaEndpoint), null);
++    const r = api.parse(USDA, { url: 'https://www.ers.usda.gov/media/x.csv', endpoint: usdaEndpoint });
++    assert.strictEqual(r.meta.format, 'csv');
++    assert.deepStrictEqual(r.items.map((i) => i.identity), [
++        'Apples | Fresh', 'Apples, applesauce | Canned', 'Bananas | Fresh', 'Grapes (raisins) | Dried', 'Oranges | Fresh', 'Strawberries | Fresh',
++    ]);
++    assert.deepStrictEqual([r.items[0].title, r.items[0].fields.name, r.items[0].fields.form, r.items[0].fields.price, r.items[0].fields.unit], [
++        'Apples', 'Apples', 'Fresh', '1.86093095', 'per pound',
++    ]);
++    assert.strictEqual(r.items[1].fields.name, 'Apples, applesauce', 'the quoted comma is part of the value');
++    assert.strictEqual(r.items[3].fields.price, '4.328343983', 'a trailing empty column is kept as the empty string');
++    assert.strictEqual(r.skipped, 1, 'the broken row');
++    assert.strictEqual(r.meta.filtered, 0);
++});
++
++t('a row missing an identity column is skipped; the first duplicate identity stays', () => {
++    const body = 'Fruit,Form,AverageRetailPrice\nApples,Fresh,1.86\nApples,Fresh,9.99\n,Fresh,1.00\nApples,,1.00\n';
++    const r = api.parse(body, { url: 'https://x.example/x.csv', endpoint: usdaEndpoint });
++    assert.deepStrictEqual(r.items.map((i) => [i.identity, i.fields.price]), [['Apples | Fresh', '1.86']]);
++    assert.strictEqual(r.skipped, 3, 'the duplicate and the two rows without a full identity');
++    assert.strictEqual(api.checkMapping({ ...usdaEndpoint, duplicates: 'last' }), null);
++});
++
++t('mapping rules: csv is an api format, an identity must be named, and identity_fields has bounds', () => {
++    assert.strictEqual(api.checkMapping({ ...usdaEndpoint, format: 'tsv' }), null, 'tsv is unchanged');
++    assert.match(api.checkMapping({ format: 'yaml', fields: { identity: 'x' } }), /format must be json, xml, tsv or csv/);
++    assert.match(api.checkMapping({ ...usdaEndpoint, items_path: 'rows' }), /items_path is ""/);
++    assert.match(api.checkMapping({ format: 'csv', identity_fields: ['A'] }), /2 to 8 paths/);
++    assert.match(api.checkMapping({ format: 'csv', identity_fields: ['A', 'B'], fields: { identity: 'A' } }), /cannot both be given/);
++    assert.match(api.checkMapping({ format: 'csv', fields: {} }), /fields\.identity, fields\.url or identity_fields is required/);
++    assert.match(api.checkMapping({ ...usdaEndpoint, items_shape: 'map' }), /identity_fields requires items_shape "array"/);
++    assert.match(api.checkMapping({ ...usdaEndpoint, identity_fields: ['Fruit', '$key'] }), /identity_fields: \$key requires items_shape "map"/);
++    assert.throws(() => api.parse('a,b\n"unterminated,1', { url: 'https://x.example/x.csv', endpoint: usdaEndpoint }), /ends inside a quoted field/);
++});
++
++t('a quoted field may hold a line break and doubled quotes', () => {
++    const body = 'Name,Form,Price\n"Apples, ""gala""",Fresh,"1.50"\n"Pears\n(anjou)",Fresh,1.20\n';
++    const r = api.parse(body, { url: 'https://x.example/x.csv', endpoint: { format: 'csv', identity_fields: ['Name', 'Form'], fields: { title: 'Name' }, extra: { price: 'Price' } } });
++    assert.deepStrictEqual(r.items.map((i) => [i.identity, i.fields.price]), [['Apples, "gala" | Fresh', '1.50'], ['Pears\n(anjou) | Fresh', '1.20']]);
++});
++
++t.run();
ddiff --git a/test/api-tsv.test.js b/test/api-tsv.test.js
iindex d8ed657..1d08b60 100644
---- a/test/api-tsv.test.js
++++ b/test/api-tsv.test.js
@@@ -74,7 +74,7 @@ t('filter: the BLS CPI rent seed keeps one series\' rows, one item per month, by
  });
  
  t('mapping and body errors', () => {
--    assert.match(api.checkMapping({ ...blsEndpoint, format: 'csv' }), /format must be json, xml or tsv/);
++    assert.match(api.checkMapping({ ...blsEndpoint, format: 'yaml' }), /format must be json, xml, tsv or csv/);
      assert.match(api.checkMapping({ ...blsEndpoint, items_path: 'rows' }), /items_path is ""/);
      assert.match(api.checkMapping({ ...blsEndpoint, duplicates: 'newest' }), /duplicates must be first or last/);
      assert.throws(() => api.parse('', { url: 'https://x.example/', endpoint: blsEndpoint }), /no header line/);