openvibers/openvibe.sources Commits
Sources service: source registry, ingestion workers with robots.txt, rate limits, conditional GET, address guard and explicit fetch-run states; RSS/Atom, sitemap, JSON-LD, mapped-API and manual adapters; items with provenance and revisions; events through an outbox
Changes
nnew file mode 100644@@@ -0,0 +1,12 @@++node_modules/++data/++dist/++.env++.env.*++!.env.example++*.db++*.db-wal++*.db-shm++*.log++*.pem++*.keynnew file mode 100644@@@ -0,0 +1,661 @@++ GNU AFFERO GENERAL PUBLIC LICENSE++ Version 3, 19 November 2007++++ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>++ Everyone is permitted to copy and distribute verbatim copies++ of this license document, but changing it is not allowed.++++ Preamble++++ The GNU Affero General Public License is a free, copyleft license for++software and other kinds of works, specifically designed to ensure++cooperation with the community in the case of network server software.++++ The licenses for most software and other practical works are designed++to take away your freedom to share and change the works. By contrast,++our General Public Licenses are intended to guarantee your freedom to++share and change all versions of a program--to make sure it remains free++software for all its users.++++ When we speak of free software, we are referring to freedom, not++price. Our General Public Licenses are designed to make sure that you++have the freedom to distribute copies of free software (and charge for++them if you wish), that you receive source code or can get it if you++want it, that you can change the software or use pieces of it in new++free programs, and that you know you can do these things.++++ Developers that use our General Public Licenses protect your rights++with two steps: (1) assert copyright on the software, and (2) offer++you this License which gives you legal permission to copy, distribute++and/or modify the software.++++ A secondary benefit of defending all users' freedom is that++improvements made in alternate versions of the program, if they++receive widespread use, become available for other developers to++incorporate. Many developers of free software are heartened and++encouraged by the resulting cooperation. However, in the case of++software used on network servers, this result may fail to come about.++The GNU General Public License permits making a modified version and++letting the public access it on a server without ever releasing its++source code to the public.++++ The GNU Affero General Public License is designed specifically to++ensure that, in such cases, the modified source code becomes available++to the community. It requires the operator of a network server to++provide the source code of the modified version running there to the++users of that server. Therefore, public use of a modified version, on++a publicly accessible server, gives the public access to the source++code of the modified version.++++ An older license, called the Affero General Public License and++published by Affero, was designed to accomplish similar goals. This is++a different license, not a version of the Affero GPL, but Affero has++released a new version of the Affero GPL which permits relicensing under++this license.++++ The precise terms and conditions for copying, distribution and++modification follow.++++ TERMS AND CONDITIONS++++ 0. Definitions.++++ "This License" refers to version 3 of the GNU Affero General Public License.++++ "Copyright" also means copyright-like laws that apply to other kinds of++works, such as semiconductor masks.++++ "The Program" refers to any copyrightable work licensed under this++License. Each licensee is addressed as "you". "Licensees" and++"recipients" may be individuals or organizations.++++ To "modify" a work means to copy from or adapt all or part of the work++in a fashion requiring copyright permission, other than the making of an++exact copy. The resulting work is called a "modified version" of the++earlier work or a work "based on" the earlier work.++++ A "covered work" means either the unmodified Program or a work based++on the Program.++++ To "propagate" a work means to do anything with it that, without++permission, would make you directly or secondarily liable for++infringement under applicable copyright law, except executing it on a++computer or modifying a private copy. Propagation includes copying,++distribution (with or without modification), making available to the++public, and in some countries other activities as well.++++ To "convey" a work means any kind of propagation that enables other++parties to make or receive copies. Mere interaction with a user through++a computer network, with no transfer of a copy, is not conveying.++++ An interactive user interface displays "Appropriate Legal Notices"++to the extent that it includes a convenient and prominently visible++feature that (1) displays an appropriate copyright notice, and (2)++tells the user that there is no warranty for the work (except to the++extent that warranties are provided), that licensees may convey the++work under this License, and how to view a copy of this License. If++the interface presents a list of user commands or options, such as a++menu, a prominent item in the list meets this criterion.++++ 1. Source Code.++++ The "source code" for a work means the preferred form of the work++for making modifications to it. "Object code" means any non-source++form of a work.++++ A "Standard Interface" means an interface that either is an official++standard defined by a recognized standards body, or, in the case of++interfaces specified for a particular programming language, one that++is widely used among developers working in that language.++++ The "System Libraries" of an executable work include anything, other++than the work as a whole, that (a) is included in the normal form of++packaging a Major Component, but which is not part of that Major++Component, and (b) serves only to enable use of the work with that++Major Component, or to implement a Standard Interface for which an++implementation is available to the public in source code form. A++"Major Component", in this context, means a major essential component++(kernel, window system, and so on) of the specific operating system++(if any) on which the executable work runs, or a compiler used to++produce the work, or an object code interpreter used to run it.++++ The "Corresponding Source" for a work in object code form means all++the source code needed to generate, install, and (for an executable++work) run the object code and to modify the work, including scripts to++control those activities. However, it does not include the work's++System Libraries, or general-purpose tools or generally available free++programs which are used unmodified in performing those activities but++which are not part of the work. For example, Corresponding Source++includes interface definition files associated with source files for++the work, and the source code for shared libraries and dynamically++linked subprograms that the work is specifically designed to require,++such as by intimate data communication or control flow between those++subprograms and other parts of the work.++++ The Corresponding Source need not include anything that users++can regenerate automatically from other parts of the Corresponding++Source.++++ The Corresponding Source for a work in source code form is that++same work.++++ 2. Basic Permissions.++++ All rights granted under this License are granted for the term of++copyright on the Program, and are irrevocable provided the stated++conditions are met. This License explicitly affirms your unlimited++permission to run the unmodified Program. The output from running a++covered work is covered by this License only if the output, given its++content, constitutes a covered work. This License acknowledges your++rights of fair use or other equivalent, as provided by copyright law.++++ You may make, run and propagate covered works that you do not++convey, without conditions so long as your license otherwise remains++in force. You may convey covered works to others for the sole purpose++of having them make modifications exclusively for you, or provide you++with facilities for running those works, provided that you comply with++the terms of this License in conveying all material for which you do++not control copyright. Those thus making or running the covered works++for you must do so exclusively on your behalf, under your direction++and control, on terms that prohibit them from making any copies of++your copyrighted material outside their relationship with you.++++ Conveying under any other circumstances is permitted solely under++the conditions stated below. Sublicensing is not allowed; section 10++makes it unnecessary.++++ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.++++ No covered work shall be deemed part of an effective technological++measure under any applicable law fulfilling obligations under article++11 of the WIPO copyright treaty adopted on 20 December 1996, or++similar laws prohibiting or restricting circumvention of such++measures.++++ When you convey a covered work, you waive any legal power to forbid++circumvention of technological measures to the extent such circumvention++is effected by exercising rights under this License with respect to++the covered work, and you disclaim any intention to limit operation or++modification of the work as a means of enforcing, against the work's++users, your or third parties' legal rights to forbid circumvention of++technological measures.++++ 4. Conveying Verbatim Copies.++++ You may convey verbatim copies of the Program's source code as you++receive it, in any medium, provided that you conspicuously and++appropriately publish on each copy an appropriate copyright notice;++keep intact all notices stating that this License and any++non-permissive terms added in accord with section 7 apply to the code;++keep intact all notices of the absence of any warranty; and give all++recipients a copy of this License along with the Program.++++ You may charge any price or no price for each copy that you convey,++and you may offer support or warranty protection for a fee.++++ 5. Conveying Modified Source Versions.++++ You may convey a work based on the Program, or the modifications to++produce it from the Program, in the form of source code under the++terms of section 4, provided that you also meet all of these conditions:++++ a) The work must carry prominent notices stating that you modified++ it, and giving a relevant date.++++ b) The work must carry prominent notices stating that it is++ released under this License and any conditions added under section++ 7. This requirement modifies the requirement in section 4 to++ "keep intact all notices".++++ c) You must license the entire work, as a whole, under this++ License to anyone who comes into possession of a copy. This++ License will therefore apply, along with any applicable section 7++ additional terms, to the whole of the work, and all its parts,++ regardless of how they are packaged. This License gives no++ permission to license the work in any other way, but it does not++ invalidate such permission if you have separately received it.++++ d) If the work has interactive user interfaces, each must display++ Appropriate Legal Notices; however, if the Program has interactive++ interfaces that do not display Appropriate Legal Notices, your++ work need not make them do so.++++ A compilation of a covered work with other separate and independent++works, which are not by their nature extensions of the covered work,++and which are not combined with it such as to form a larger program,++in or on a volume of a storage or distribution medium, is called an++"aggregate" if the compilation and its resulting copyright are not++used to limit the access or legal rights of the compilation's users++beyond what the individual works permit. Inclusion of a covered work++in an aggregate does not cause this License to apply to the other++parts of the aggregate.++++ 6. Conveying Non-Source Forms.++++ You may convey a covered work in object code form under the terms++of sections 4 and 5, provided that you also convey the++machine-readable Corresponding Source under the terms of this License,++in one of these ways:++++ a) Convey the object code in, or embodied in, a physical product++ (including a physical distribution medium), accompanied by the++ Corresponding Source fixed on a durable physical medium++ customarily used for software interchange.++++ b) Convey the object code in, or embodied in, a physical product++ (including a physical distribution medium), accompanied by a++ written offer, valid for at least three years and valid for as++ long as you offer spare parts or customer support for that product++ model, to give anyone who possesses the object code either (1) a++ copy of the Corresponding Source for all the software in the++ product that is covered by this License, on a durable physical++ medium customarily used for software interchange, for a price no++ more than your reasonable cost of physically performing this++ conveying of source, or (2) access to copy the++ Corresponding Source from a network server at no charge.++++ c) Convey individual copies of the object code with a copy of the++ written offer to provide the Corresponding Source. This++ alternative is allowed only occasionally and noncommercially, and++ only if you received the object code with such an offer, in accord++ with subsection 6b.++++ d) Convey the object code by offering access from a designated++ place (gratis or for a charge), and offer equivalent access to the++ Corresponding Source in the same way through the same place at no++ further charge. You need not require recipients to copy the++ Corresponding Source along with the object code. If the place to++ copy the object code is a network server, the Corresponding Source++ may be on a different server (operated by you or a third party)++ that supports equivalent copying facilities, provided you maintain++ clear directions next to the object code saying where to find the++ Corresponding Source. Regardless of what server hosts the++ Corresponding Source, you remain obligated to ensure that it is++ available for as long as needed to satisfy these requirements.++++ e) Convey the object code using peer-to-peer transmission, provided++ you inform other peers where the object code and Corresponding++ Source of the work are being offered to the general public at no++ charge under subsection 6d.++++ A separable portion of the object code, whose source code is excluded++from the Corresponding Source as a System Library, need not be++included in conveying the object code work.++++ A "User Product" is either (1) a "consumer product", which means any++tangible personal property which is normally used for personal, family,++or household purposes, or (2) anything designed or sold for incorporation++into a dwelling. In determining whether a product is a consumer product,++doubtful cases shall be resolved in favor of coverage. For a particular++product received by a particular user, "normally used" refers to a++typical or common use of that class of product, regardless of the status++of the particular user or of the way in which the particular user++actually uses, or expects or is expected to use, the product. A product++is a consumer product regardless of whether the product has substantial++commercial, industrial or non-consumer uses, unless such uses represent++the only significant mode of use of the product.++++ "Installation Information" for a User Product means any methods,++procedures, authorization keys, or other information required to install++and execute modified versions of a covered work in that User Product from++a modified version of its Corresponding Source. The information must++suffice to ensure that the continued functioning of the modified object++code is in no case prevented or interfered with solely because++modification has been made.++++ If you convey an object code work under this section in, or with, or++specifically for use in, a User Product, and the conveying occurs as++part of a transaction in which the right of possession and use of the++User Product is transferred to the recipient in perpetuity or for a++fixed term (regardless of how the transaction is characterized), the++Corresponding Source conveyed under this section must be accompanied++by the Installation Information. But this requirement does not apply++if neither you nor any third party retains the ability to install++modified object code on the User Product (for example, the work has++been installed in ROM).++++ The requirement to provide Installation Information does not include a++requirement to continue to provide support service, warranty, or updates++for a work that has been modified or installed by the recipient, or for++the User Product in which it has been modified or installed. Access to a++network may be denied when the modification itself materially and++adversely affects the operation of the network or violates the rules and++protocols for communication across the network.++++ Corresponding Source conveyed, and Installation Information provided,++in accord with this section must be in a format that is publicly++documented (and with an implementation available to the public in++source code form), and must require no special password or key for++unpacking, reading or copying.++++ 7. Additional Terms.++++ "Additional permissions" are terms that supplement the terms of this++License by making exceptions from one or more of its conditions.++Additional permissions that are applicable to the entire Program shall++be treated as though they were included in this License, to the extent++that they are valid under applicable law. If additional permissions++apply only to part of the Program, that part may be used separately++under those permissions, but the entire Program remains governed by++this License without regard to the additional permissions.++++ When you convey a copy of a covered work, you may at your option++remove any additional permissions from that copy, or from any part of++it. (Additional permissions may be written to require their own++removal in certain cases when you modify the work.) You may place++additional permissions on material, added by you to a covered work,++for which you have or can give appropriate copyright permission.++++ Notwithstanding any other provision of this License, for material you++add to a covered work, you may (if authorized by the copyright holders of++that material) supplement the terms of this License with terms:++++ a) Disclaiming warranty or limiting liability differently from the++ terms of sections 15 and 16 of this License; or++++ b) Requiring preservation of specified reasonable legal notices or++ author attributions in that material or in the Appropriate Legal++ Notices displayed by works containing it; or++++ c) Prohibiting misrepresentation of the origin of that material, or++ requiring that modified versions of such material be marked in++ reasonable ways as different from the original version; or++++ d) Limiting the use for publicity purposes of names of licensors or++ authors of the material; or++++ e) Declining to grant rights under trademark law for use of some++ trade names, trademarks, or service marks; or++++ f) Requiring indemnification of licensors and authors of that++ material by anyone who conveys the material (or modified versions of++ it) with contractual assumptions of liability to the recipient, for++ any liability that these contractual assumptions directly impose on++ those licensors and authors.++++ All other non-permissive additional terms are considered "further++restrictions" within the meaning of section 10. If the Program as you++received it, or any part of it, contains a notice stating that it is++governed by this License along with a term that is a further++restriction, you may remove that term. If a license document contains++a further restriction but permits relicensing or conveying under this++License, you may add to a covered work material governed by the terms++of that license document, provided that the further restriction does++not survive such relicensing or conveying.++++ If you add terms to a covered work in accord with this section, you++must place, in the relevant source files, a statement of the++additional terms that apply to those files, or a notice indicating++where to find the applicable terms.++++ Additional terms, permissive or non-permissive, may be stated in the++form of a separately written license, or stated as exceptions;++the above requirements apply either way.++++ 8. Termination.++++ You may not propagate or modify a covered work except as expressly++provided under this License. Any attempt otherwise to propagate or++modify it is void, and will automatically terminate your rights under++this License (including any patent licenses granted under the third++paragraph of section 11).++++ However, if you cease all violation of this License, then your++license from a particular copyright holder is reinstated (a)++provisionally, unless and until the copyright holder explicitly and++finally terminates your license, and (b) permanently, if the copyright++holder fails to notify you of the violation by some reasonable means++prior to 60 days after the cessation.++++ Moreover, your license from a particular copyright holder is++reinstated permanently if the copyright holder notifies you of the++violation by some reasonable means, this is the first time you have++received notice of violation of this License (for any work) from that++copyright holder, and you cure the violation prior to 30 days after++your receipt of the notice.++++ Termination of your rights under this section does not terminate the++licenses of parties who have received copies or rights from you under++this License. If your rights have been terminated and not permanently++reinstated, you do not qualify to receive new licenses for the same++material under section 10.++++ 9. Acceptance Not Required for Having Copies.++++ You are not required to accept this License in order to receive or++run a copy of the Program. Ancillary propagation of a covered work++occurring solely as a consequence of using peer-to-peer transmission++to receive a copy likewise does not require acceptance. However,++nothing other than this License grants you permission to propagate or++modify any covered work. These actions infringe copyright if you do++not accept this License. Therefore, by modifying or propagating a++covered work, you indicate your acceptance of this License to do so.++++ 10. Automatic Licensing of Downstream Recipients.++++ Each time you convey a covered work, the recipient automatically++receives a license from the original licensors, to run, modify and++propagate that work, subject to this License. You are not responsible++for enforcing compliance by third parties with this License.++++ An "entity transaction" is a transaction transferring control of an++organization, or substantially all assets of one, or subdividing an++organization, or merging organizations. If propagation of a covered++work results from an entity transaction, each party to that++transaction who receives a copy of the work also receives whatever++licenses to the work the party's predecessor in interest had or could++give under the previous paragraph, plus a right to possession of the++Corresponding Source of the work from the predecessor in interest, if++the predecessor has it or can get it with reasonable efforts.++++ You may not impose any further restrictions on the exercise of the++rights granted or affirmed under this License. For example, you may++not impose a license fee, royalty, or other charge for exercise of++rights granted under this License, and you may not initiate litigation++(including a cross-claim or counterclaim in a lawsuit) alleging that++any patent claim is infringed by making, using, selling, offering for++sale, or importing the Program or any portion of it.++++ 11. Patents.++++ A "contributor" is a copyright holder who authorizes use under this++License of the Program or a work on which the Program is based. The++work thus licensed is called the contributor's "contributor version".++++ A contributor's "essential patent claims" are all patent claims++owned or controlled by the contributor, whether already acquired or++hereafter acquired, that would be infringed by some manner, permitted++by this License, of making, using, or selling its contributor version,++but do not include claims that would be infringed only as a++consequence of further modification of the contributor version. For++purposes of this definition, "control" includes the right to grant++patent sublicenses in a manner consistent with the requirements of++this License.++++ Each contributor grants you a non-exclusive, worldwide, royalty-free++patent license under the contributor's essential patent claims, to++make, use, sell, offer for sale, import and otherwise run, modify and++propagate the contents of its contributor version.++++ In the following three paragraphs, a "patent license" is any express++agreement or commitment, however denominated, not to enforce a patent++(such as an express permission to practice a patent or covenant not to++sue for patent infringement). To "grant" such a patent license to a++party means to make such an agreement or commitment not to enforce a++patent against the party.++++ If you convey a covered work, knowingly relying on a patent license,++and the Corresponding Source of the work is not available for anyone++to copy, free of charge and under the terms of this License, through a++publicly available network server or other readily accessible means,++then you must either (1) cause the Corresponding Source to be so++available, or (2) arrange to deprive yourself of the benefit of the++patent license for this particular work, or (3) arrange, in a manner++consistent with the requirements of this License, to extend the patent++license to downstream recipients. "Knowingly relying" means you have++actual knowledge that, but for the patent license, your conveying the++covered work in a country, or your recipient's use of the covered work++in a country, would infringe one or more identifiable patents in that++country that you have reason to believe are valid.++++ If, pursuant to or in connection with a single transaction or++arrangement, you convey, or propagate by procuring conveyance of, a++covered work, and grant a patent license to some of the parties++receiving the covered work authorizing them to use, propagate, modify++or convey a specific copy of the covered work, then the patent license++you grant is automatically extended to all recipients of the covered++work and works based on it.++++ A patent license is "discriminatory" if it does not include within++the scope of its coverage, prohibits the exercise of, or is++conditioned on the non-exercise of one or more of the rights that are++specifically granted under this License. You may not convey a covered++work if you are a party to an arrangement with a third party that is++in the business of distributing software, under which you make payment++to the third party based on the extent of your activity of conveying++the work, and under which the third party grants, to any of the++parties who would receive the covered work from you, a discriminatory++patent license (a) in connection with copies of the covered work++conveyed by you (or copies made from those copies), or (b) primarily++for and in connection with specific products or compilations that++contain the covered work, unless you entered into that arrangement,++or that patent license was granted, prior to 28 March 2007.++++ Nothing in this License shall be construed as excluding or limiting++any implied license or other defenses to infringement that may++otherwise be available to you under applicable patent law.++++ 12. No Surrender of Others' Freedom.++++ If conditions are imposed on you (whether by court order, agreement or++otherwise) that contradict the conditions of this License, they do not++excuse you from the conditions of this License. If you cannot convey a++covered work so as to satisfy simultaneously your obligations under this++License and any other pertinent obligations, then as a consequence you may++not convey it at all. For example, if you agree to terms that obligate you++to collect a royalty for further conveying from those to whom you convey++the Program, the only way you could satisfy both those terms and this++License would be to refrain entirely from conveying the Program.++++ 13. Remote Network Interaction; Use with the GNU General Public License.++++ Notwithstanding any other provision of this License, if you modify the++Program, your modified version must prominently offer all users++interacting with it remotely through a computer network (if your version++supports such interaction) an opportunity to receive the Corresponding++Source of your version by providing access to the Corresponding Source++from a network server at no charge, through some standard or customary++means of facilitating copying of software. This Corresponding Source++shall include the Corresponding Source for any work covered by version 3++of the GNU General Public License that is incorporated pursuant to the++following paragraph.++++ Notwithstanding any other provision of this License, you have++permission to link or combine any covered work with a work licensed++under version 3 of the GNU General Public License into a single++combined work, and to convey the resulting work. The terms of this++License will continue to apply to the part which is the covered work,++but the work with which it is combined will remain governed by version++3 of the GNU General Public License.++++ 14. Revised Versions of this License.++++ The Free Software Foundation may publish revised and/or new versions of++the GNU Affero General Public License from time to time. Such new versions++will be similar in spirit to the present version, but may differ in detail to++address new problems or concerns.++++ Each version is given a distinguishing version number. If the++Program specifies that a certain numbered version of the GNU Affero General++Public License "or any later version" applies to it, you have the++option of following the terms and conditions either of that numbered++version or of any later version published by the Free Software++Foundation. If the Program does not specify a version number of the++GNU Affero General Public License, you may choose any version ever published++by the Free Software Foundation.++++ If the Program specifies that a proxy can decide which future++versions of the GNU Affero General Public License can be used, that proxy's++public statement of acceptance of a version permanently authorizes you++to choose that version for the Program.++++ Later license versions may give you additional or different++permissions. However, no additional obligations are imposed on any++author or copyright holder as a result of your choosing to follow a++later version.++++ 15. Disclaimer of Warranty.++++ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY++APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT++HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY++OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,++THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR++PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM++IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF++ALL NECESSARY SERVICING, REPAIR OR CORRECTION.++++ 16. Limitation of Liability.++++ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING++WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS++THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY++GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE++USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF++DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD++PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),++EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF++SUCH DAMAGES.++++ 17. Interpretation of Sections 15 and 16.++++ If the disclaimer of warranty and limitation of liability provided++above cannot be given local legal effect according to their terms,++reviewing courts shall apply local law that most closely approximates++an absolute waiver of all civil liability in connection with the++Program, unless a warranty or assumption of liability accompanies a++copy of the Program in return for a fee.++++ END OF TERMS AND CONDITIONS++++ How to Apply These Terms to Your New Programs++++ If you develop a new program, and you want it to be of the greatest++possible use to the public, the best way to achieve this is to make it++free software which everyone can redistribute and change under these terms.++++ To do so, attach the following notices to the program. It is safest++to attach them to the start of each source file to most effectively++state the exclusion of warranty; and each file should have at least++the "copyright" line and a pointer to where the full notice is found.++++ <one line to give the program's name and a brief idea of what it does.>++ Copyright (C) <year> <name of author>++++ This program is free software: you can redistribute it and/or modify++ it under the terms of the GNU Affero General Public License as published++ by the Free Software Foundation, either version 3 of the License, or++ (at your option) any later version.++++ This program is distributed in the hope that it will be useful,++ but WITHOUT ANY WARRANTY; without even the implied warranty of++ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the++ GNU Affero General Public License for more details.++++ You should have received a copy of the GNU Affero General Public License++ along with this program. If not, see <https://www.gnu.org/licenses/>.++++Also add information on how to contact you by electronic and paper mail.++++ If your software can interact with users remotely through a computer++network, you should also make sure that it provides a way for users to++get its source. For example, if your program is a web application, its++interface could display a "Source" link that leads users to an archive++of the code. There are many ways you could offer source, and different++solutions will be better for different programs; see section 13 for the++specific requirements.++++ You should also get your employer (if you work as a programmer) or school,++if any, to sign a "copyright disclaimer" for the program, if necessary.++For more information on this, and how to apply and follow the GNU AGPL, see++<https://www.gnu.org/licenses/>.nnew file mode 100644@@@ -0,0 +1,1395 @@++{++ "name": "openvibe-sources",++ "version": "0.1.0",++ "lockfileVersion": 3,++ "requires": true,++ "packages": {++ "": {++ "name": "openvibe-sources",++ "version": "0.1.0",++ "license": "AGPL-3.0-only",++ "dependencies": {++ "better-sqlite3": "^11.10.0",++ "dotenv": "^16.4.7",++ "express": "^4.21.2",++ "fast-xml-parser": "^4.5.3",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0"++ },++ "devDependencies": {++ "ajv": "^8.20.0",++ "ajv-formats": "^3.0.1"++ },++ "engines": {++ "node": ">=22.0.0"++ }++ },++ "node_modules/accepts": {++ "version": "1.3.8",++ "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",++ "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",++ "license": "MIT",++ "dependencies": {++ "mime-types": "~2.1.34",++ "negotiator": "0.6.3"++ },++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/ajv": {++ "version": "8.20.0",++ "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz",++ "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==",++ "license": "MIT",++ "dependencies": {++ "fast-deep-equal": "^3.1.3",++ "fast-uri": "^3.0.1",++ "json-schema-traverse": "^1.0.0",++ "require-from-string": "^2.0.2"++ },++ "funding": {++ "type": "github",++ "url": "https://github.com/sponsors/epoberezkin"++ }++ },++ "node_modules/ajv-formats": {++ "version": "3.0.1",++ "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz",++ "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==",++ "license": "MIT",++ "dependencies": {++ "ajv": "^8.0.0"++ },++ "peerDependencies": {++ "ajv": "^8.0.0"++ },++ "peerDependenciesMeta": {++ "ajv": {++ "optional": true++ }++ }++ },++ "node_modules/array-flatten": {++ "version": "1.1.1",++ "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",++ "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",++ "license": "MIT"++ },++ "node_modules/base64-js": {++ "version": "1.5.1",++ "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",++ "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "MIT"++ },++ "node_modules/better-sqlite3": {++ "version": "11.10.0",++ "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz",++ "integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",++ "hasInstallScript": true,++ "license": "MIT",++ "dependencies": {++ "bindings": "^1.5.0",++ "prebuild-install": "^7.1.1"++ }++ },++ "node_modules/bindings": {++ "version": "1.5.0",++ "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",++ "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",++ "license": "MIT",++ "dependencies": {++ "file-uri-to-path": "1.0.0"++ }++ },++ "node_modules/bl": {++ "version": "4.1.0",++ "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",++ "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",++ "license": "MIT",++ "dependencies": {++ "buffer": "^5.5.0",++ "inherits": "^2.0.4",++ "readable-stream": "^3.4.0"++ }++ },++ "node_modules/body-parser": {++ "version": "1.20.8",++ "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",++ "integrity": "sha512-JNcyFQ64OiijEkPzUBTCe+hyPXUD/3LEldGQ6iF5LR1w00mx9o7xtDWHXBY2iItjdCFGoilOLNQbH943ut7pHA==",++ "license": "MIT",++ "dependencies": {++ "bytes": "~3.1.2",++ "content-type": "~1.0.5",++ "debug": "2.6.9",++ "depd": "2.0.0",++ "destroy": "~1.2.0",++ "http-errors": "~2.0.1",++ "iconv-lite": "~0.4.24",++ "on-finished": "~2.4.1",++ "qs": "~6.16.0",++ "raw-body": "~2.5.3",++ "type-is": "~1.6.18",++ "unpipe": "~1.0.0"++ },++ "engines": {++ "node": ">= 0.8",++ "npm": "1.2.8000 || >= 1.4.16"++ }++ },++ "node_modules/buffer": {++ "version": "5.7.1",++ "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",++ "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "MIT",++ "dependencies": {++ "base64-js": "^1.3.1",++ "ieee754": "^1.1.13"++ }++ },++ "node_modules/bytes": {++ "version": "3.1.2",++ "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",++ "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/call-bind-apply-helpers": {++ "version": "1.0.2",++ "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",++ "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",++ "license": "MIT",++ "dependencies": {++ "es-errors": "^1.3.0",++ "function-bind": "^1.1.2"++ },++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/call-bound": {++ "version": "1.0.4",++ "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",++ "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",++ "license": "MIT",++ "dependencies": {++ "call-bind-apply-helpers": "^1.0.2",++ "get-intrinsic": "^1.3.0"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/chownr": {++ "version": "1.1.4",++ "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",++ "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",++ "license": "ISC"++ },++ "node_modules/content-disposition": {++ "version": "0.5.4",++ "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",++ "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",++ "license": "MIT",++ "dependencies": {++ "safe-buffer": "5.2.1"++ },++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/content-type": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",++ "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/cookie": {++ "version": "0.7.2",++ "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",++ "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/cookie-signature": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.7.tgz",++ "integrity": "sha512-NXdYc3dLr47pBkpUCHtKSwIOQXLVn8dZEuywboCOJY/osA0wFSLlSawr3KN8qXJEyX66FcONTH8EIlVuK0yyFA==",++ "license": "MIT"++ },++ "node_modules/debug": {++ "version": "2.6.9",++ "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",++ "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",++ "license": "MIT",++ "dependencies": {++ "ms": "2.0.0"++ }++ },++ "node_modules/decompress-response": {++ "version": "6.0.0",++ "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",++ "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",++ "license": "MIT",++ "dependencies": {++ "mimic-response": "^3.1.0"++ },++ "engines": {++ "node": ">=10"++ },++ "funding": {++ "url": "https://github.com/sponsors/sindresorhus"++ }++ },++ "node_modules/deep-extend": {++ "version": "0.6.0",++ "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",++ "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",++ "license": "MIT",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/depd": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",++ "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/destroy": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",++ "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8",++ "npm": "1.2.8000 || >= 1.4.16"++ }++ },++ "node_modules/detect-libc": {++ "version": "2.1.2",++ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",++ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",++ "license": "Apache-2.0",++ "engines": {++ "node": ">=8"++ }++ },++ "node_modules/dotenv": {++ "version": "16.6.1",++ "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",++ "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==",++ "license": "BSD-2-Clause",++ "engines": {++ "node": ">=12"++ },++ "funding": {++ "url": "https://dotenvx.com"++ }++ },++ "node_modules/dunder-proto": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",++ "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",++ "license": "MIT",++ "dependencies": {++ "call-bind-apply-helpers": "^1.0.1",++ "es-errors": "^1.3.0",++ "gopd": "^1.2.0"++ },++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/ee-first": {++ "version": "1.1.1",++ "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",++ "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==",++ "license": "MIT"++ },++ "node_modules/encodeurl": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",++ "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/end-of-stream": {++ "version": "1.4.5",++ "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",++ "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",++ "license": "MIT",++ "dependencies": {++ "once": "^1.4.0"++ }++ },++ "node_modules/es-define-property": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",++ "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/es-errors": {++ "version": "1.3.0",++ "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",++ "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/es-object-atoms": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz",++ "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==",++ "license": "MIT",++ "dependencies": {++ "es-errors": "^1.3.0"++ },++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/escape-html": {++ "version": "1.0.3",++ "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",++ "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==",++ "license": "MIT"++ },++ "node_modules/etag": {++ "version": "1.8.1",++ "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",++ "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/expand-template": {++ "version": "2.0.3",++ "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",++ "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",++ "license": "(MIT OR WTFPL)",++ "engines": {++ "node": ">=6"++ }++ },++ "node_modules/express": {++ "version": "4.22.3",++ "resolved": "https://registry.npmjs.org/express/-/express-4.22.3.tgz",++ "integrity": "sha512-Bdcs4+3qlpVlx2NRn6fgX2Ue2/gGRaPeawebgclM0ERSCqDpA+owF1fdPwjJUTAJWMTuAaxjDf+hzb0/4eKvvw==",++ "license": "MIT",++ "dependencies": {++ "accepts": "~1.3.8",++ "array-flatten": "1.1.1",++ "body-parser": "~1.20.5",++ "content-disposition": "~0.5.4",++ "content-type": "~1.0.4",++ "cookie": "~0.7.1",++ "cookie-signature": "~1.0.6",++ "debug": "2.6.9",++ "depd": "2.0.0",++ "encodeurl": "~2.0.0",++ "escape-html": "~1.0.3",++ "etag": "~1.8.1",++ "finalhandler": "~1.3.1",++ "fresh": "~0.5.2",++ "http-errors": "~2.0.0",++ "merge-descriptors": "1.0.3",++ "methods": "~1.1.2",++ "on-finished": "~2.4.1",++ "parseurl": "~1.3.3",++ "path-to-regexp": "~0.1.13",++ "proxy-addr": "~2.0.7",++ "qs": "~6.16.0",++ "range-parser": "~1.2.1",++ "safe-buffer": "5.2.1",++ "send": "~0.19.0",++ "serve-static": "~1.16.2",++ "setprototypeof": "1.2.0",++ "statuses": "~2.0.1",++ "type-is": "~1.6.18",++ "utils-merge": "1.0.1",++ "vary": "~1.1.2"++ },++ "engines": {++ "node": ">= 0.10.0"++ },++ "funding": {++ "type": "opencollective",++ "url": "https://opencollective.com/express"++ }++ },++ "node_modules/fast-deep-equal": {++ "version": "3.1.3",++ "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",++ "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==",++ "license": "MIT"++ },++ "node_modules/fast-uri": {++ "version": "3.1.8",++ "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz",++ "integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/fastify"++ },++ {++ "type": "opencollective",++ "url": "https://opencollective.com/fastify"++ }++ ],++ "license": "BSD-3-Clause"++ },++ "node_modules/fast-xml-parser": {++ "version": "4.5.7",++ "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-4.5.7.tgz",++ "integrity": "sha512-a6Qh1RMCNbSrU1+sAyAAZH3rTe+OaWJbNZIq0S+ifZciUUOQtlVxBJwoTUE2bYhysmG/RYyI5WJFIKdBahJdrQ==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/NaturalIntelligence"++ }++ ],++ "license": "MIT",++ "dependencies": {++ "strnum": "^1.0.5"++ },++ "bin": {++ "fxparser": "src/cli/cli.js"++ }++ },++ "node_modules/file-uri-to-path": {++ "version": "1.0.0",++ "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",++ "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",++ "license": "MIT"++ },++ "node_modules/finalhandler": {++ "version": "1.3.2",++ "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",++ "integrity": "sha512-aA4RyPcd3badbdABGDuTXCMTtOneUCAYH/gxoYRTZlIJdF0YPWuGqiAsIrhNnnqdXGswYk6dGujem4w80UJFhg==",++ "license": "MIT",++ "dependencies": {++ "debug": "2.6.9",++ "encodeurl": "~2.0.0",++ "escape-html": "~1.0.3",++ "on-finished": "~2.4.1",++ "parseurl": "~1.3.3",++ "statuses": "~2.0.2",++ "unpipe": "~1.0.0"++ },++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/forwarded": {++ "version": "0.2.0",++ "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",++ "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/fresh": {++ "version": "0.5.2",++ "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",++ "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/fs-constants": {++ "version": "1.0.0",++ "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",++ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",++ "license": "MIT"++ },++ "node_modules/function-bind": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",++ "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",++ "license": "MIT",++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/get-intrinsic": {++ "version": "1.3.0",++ "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",++ "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",++ "license": "MIT",++ "dependencies": {++ "call-bind-apply-helpers": "^1.0.2",++ "es-define-property": "^1.0.1",++ "es-errors": "^1.3.0",++ "es-object-atoms": "^1.1.1",++ "function-bind": "^1.1.2",++ "get-proto": "^1.0.1",++ "gopd": "^1.2.0",++ "has-symbols": "^1.1.0",++ "hasown": "^2.0.2",++ "math-intrinsics": "^1.1.0"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/get-proto": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",++ "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",++ "license": "MIT",++ "dependencies": {++ "dunder-proto": "^1.0.1",++ "es-object-atoms": "^1.0.0"++ },++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/github-from-package": {++ "version": "0.0.0",++ "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",++ "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",++ "license": "MIT"++ },++ "node_modules/gopd": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",++ "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/has-symbols": {++ "version": "1.1.0",++ "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",++ "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/hasown": {++ "version": "2.0.4",++ "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",++ "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",++ "license": "MIT",++ "dependencies": {++ "function-bind": "^1.1.2"++ },++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/http-errors": {++ "version": "2.0.1",++ "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",++ "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==",++ "license": "MIT",++ "dependencies": {++ "depd": "~2.0.0",++ "inherits": "~2.0.4",++ "setprototypeof": "~1.2.0",++ "statuses": "~2.0.2",++ "toidentifier": "~1.0.1"++ },++ "engines": {++ "node": ">= 0.8"++ },++ "funding": {++ "type": "opencollective",++ "url": "https://opencollective.com/express"++ }++ },++ "node_modules/iconv-lite": {++ "version": "0.4.24",++ "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",++ "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",++ "license": "MIT",++ "dependencies": {++ "safer-buffer": ">= 2.1.2 < 3"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/ieee754": {++ "version": "1.2.1",++ "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",++ "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "BSD-3-Clause"++ },++ "node_modules/inherits": {++ "version": "2.0.4",++ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",++ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",++ "license": "ISC"++ },++ "node_modules/ini": {++ "version": "1.3.8",++ "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",++ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",++ "license": "ISC"++ },++ "node_modules/ipaddr.js": {++ "version": "1.9.1",++ "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",++ "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.10"++ }++ },++ "node_modules/json-schema-traverse": {++ "version": "1.0.0",++ "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz",++ "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==",++ "license": "MIT"++ },++ "node_modules/math-intrinsics": {++ "version": "1.1.0",++ "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",++ "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ }++ },++ "node_modules/media-typer": {++ "version": "0.3.0",++ "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",++ "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/merge-descriptors": {++ "version": "1.0.3",++ "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",++ "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",++ "license": "MIT",++ "funding": {++ "url": "https://github.com/sponsors/sindresorhus"++ }++ },++ "node_modules/methods": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",++ "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/mime": {++ "version": "1.6.0",++ "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",++ "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",++ "license": "MIT",++ "bin": {++ "mime": "cli.js"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/mime-db": {++ "version": "1.52.0",++ "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",++ "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/mime-types": {++ "version": "2.1.35",++ "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",++ "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",++ "license": "MIT",++ "dependencies": {++ "mime-db": "1.52.0"++ },++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/mimic-response": {++ "version": "3.1.0",++ "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",++ "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",++ "license": "MIT",++ "engines": {++ "node": ">=10"++ },++ "funding": {++ "url": "https://github.com/sponsors/sindresorhus"++ }++ },++ "node_modules/minimist": {++ "version": "1.2.8",++ "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",++ "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",++ "license": "MIT",++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/mkdirp-classic": {++ "version": "0.5.3",++ "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",++ "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",++ "license": "MIT"++ },++ "node_modules/ms": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",++ "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",++ "license": "MIT"++ },++ "node_modules/napi-build-utils": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",++ "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",++ "license": "MIT"++ },++ "node_modules/negotiator": {++ "version": "0.6.3",++ "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",++ "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/node-abi": {++ "version": "3.96.0",++ "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",++ "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",++ "license": "MIT",++ "dependencies": {++ "semver": "^7.3.5"++ },++ "engines": {++ "node": ">=10"++ }++ },++ "node_modules/object-inspect": {++ "version": "1.13.4",++ "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",++ "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/on-finished": {++ "version": "2.4.1",++ "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",++ "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",++ "license": "MIT",++ "dependencies": {++ "ee-first": "1.1.1"++ },++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/once": {++ "version": "1.4.0",++ "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",++ "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",++ "license": "ISC",++ "dependencies": {++ "wrappy": "1"++ }++ },++ "node_modules/openvibe-contracts": {++ "version": "0.7.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0",++ "integrity": "sha512-gUrjxbTy/HX0QMQGoDPphmFdwm7iitmNMMchs9tDSDGR47FRFHL/RgBgIKYR1S7M6ICu261YycFV8+YVglafXA==",++ "license": "AGPL-3.0-only",++ "dependencies": {++ "ajv": "^8.17.1",++ "ajv-formats": "^3.0.1"++ },++ "bin": {++ "openvibe-contracts-check": "bin/check-service.js"++ },++ "engines": {++ "node": ">=20"++ }++ },++ "node_modules/parseurl": {++ "version": "1.3.3",++ "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",++ "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/path-to-regexp": {++ "version": "0.1.13",++ "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.13.tgz",++ "integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",++ "license": "MIT"++ },++ "node_modules/prebuild-install": {++ "version": "7.1.3",++ "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",++ "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",++ "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",++ "license": "MIT",++ "dependencies": {++ "detect-libc": "^2.0.0",++ "expand-template": "^2.0.3",++ "github-from-package": "0.0.0",++ "minimist": "^1.2.3",++ "mkdirp-classic": "^0.5.3",++ "napi-build-utils": "^2.0.0",++ "node-abi": "^3.3.0",++ "pump": "^3.0.0",++ "rc": "^1.2.7",++ "simple-get": "^4.0.0",++ "tar-fs": "^2.0.0",++ "tunnel-agent": "^0.6.0"++ },++ "bin": {++ "prebuild-install": "bin.js"++ },++ "engines": {++ "node": ">=10"++ }++ },++ "node_modules/proxy-addr": {++ "version": "2.0.8",++ "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",++ "integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==",++ "license": "MIT",++ "dependencies": {++ "forwarded": "0.2.0",++ "ipaddr.js": "1.9.1"++ },++ "engines": {++ "node": ">= 0.10"++ },++ "funding": {++ "type": "opencollective",++ "url": "https://opencollective.com/express"++ }++ },++ "node_modules/pump": {++ "version": "3.0.4",++ "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",++ "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",++ "license": "MIT",++ "dependencies": {++ "end-of-stream": "^1.1.0",++ "once": "^1.3.1"++ }++ },++ "node_modules/qs": {++ "version": "6.16.0",++ "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",++ "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",++ "license": "BSD-3-Clause",++ "dependencies": {++ "es-define-property": "^1.0.1",++ "side-channel": "^1.1.1"++ },++ "engines": {++ "node": ">=0.6"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/range-parser": {++ "version": "1.2.1",++ "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",++ "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/raw-body": {++ "version": "2.5.3",++ "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz",++ "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==",++ "license": "MIT",++ "dependencies": {++ "bytes": "~3.1.2",++ "http-errors": "~2.0.1",++ "iconv-lite": "~0.4.24",++ "unpipe": "~1.0.0"++ },++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/rc": {++ "version": "1.2.8",++ "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",++ "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",++ "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",++ "dependencies": {++ "deep-extend": "^0.6.0",++ "ini": "~1.3.0",++ "minimist": "^1.2.0",++ "strip-json-comments": "~2.0.1"++ },++ "bin": {++ "rc": "cli.js"++ }++ },++ "node_modules/readable-stream": {++ "version": "3.6.2",++ "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",++ "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",++ "license": "MIT",++ "dependencies": {++ "inherits": "^2.0.3",++ "string_decoder": "^1.1.1",++ "util-deprecate": "^1.0.1"++ },++ "engines": {++ "node": ">= 6"++ }++ },++ "node_modules/require-from-string": {++ "version": "2.0.2",++ "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",++ "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/safe-buffer": {++ "version": "5.2.1",++ "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",++ "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "MIT"++ },++ "node_modules/safer-buffer": {++ "version": "2.1.2",++ "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",++ "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",++ "license": "MIT"++ },++ "node_modules/semver": {++ "version": "7.8.5",++ "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz",++ "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==",++ "license": "ISC",++ "bin": {++ "semver": "bin/semver.js"++ },++ "engines": {++ "node": ">=10"++ }++ },++ "node_modules/send": {++ "version": "0.19.2",++ "resolved": "https://registry.npmjs.org/send/-/send-0.19.2.tgz",++ "integrity": "sha512-VMbMxbDeehAxpOtWJXlcUS5E8iXh6QmN+BkRX1GARS3wRaXEEgzCcB10gTQazO42tpNIya8xIyNx8fll1OFPrg==",++ "license": "MIT",++ "dependencies": {++ "debug": "2.6.9",++ "depd": "2.0.0",++ "destroy": "1.2.0",++ "encodeurl": "~2.0.0",++ "escape-html": "~1.0.3",++ "etag": "~1.8.1",++ "fresh": "~0.5.2",++ "http-errors": "~2.0.1",++ "mime": "1.6.0",++ "ms": "2.1.3",++ "on-finished": "~2.4.1",++ "range-parser": "~1.2.1",++ "statuses": "~2.0.2"++ },++ "engines": {++ "node": ">= 0.8.0"++ }++ },++ "node_modules/send/node_modules/ms": {++ "version": "2.1.3",++ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",++ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",++ "license": "MIT"++ },++ "node_modules/serve-static": {++ "version": "1.16.3",++ "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz",++ "integrity": "sha512-x0RTqQel6g5SY7Lg6ZreMmsOzncHFU7nhnRWkKgWuMTu5NN0DR5oruckMqRvacAN9d5w6ARnRBXl9xhDCgfMeA==",++ "license": "MIT",++ "dependencies": {++ "encodeurl": "~2.0.0",++ "escape-html": "~1.0.3",++ "parseurl": "~1.3.3",++ "send": "~0.19.1"++ },++ "engines": {++ "node": ">= 0.8.0"++ }++ },++ "node_modules/setprototypeof": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",++ "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",++ "license": "ISC"++ },++ "node_modules/side-channel": {++ "version": "1.1.1",++ "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",++ "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",++ "license": "MIT",++ "dependencies": {++ "es-errors": "^1.3.0",++ "object-inspect": "^1.13.4",++ "side-channel-list": "^1.0.1",++ "side-channel-map": "^1.0.1",++ "side-channel-weakmap": "^1.0.2"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/side-channel-list": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",++ "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",++ "license": "MIT",++ "dependencies": {++ "es-errors": "^1.3.0",++ "object-inspect": "^1.13.4"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/side-channel-map": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",++ "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",++ "license": "MIT",++ "dependencies": {++ "call-bound": "^1.0.2",++ "es-errors": "^1.3.0",++ "get-intrinsic": "^1.2.5",++ "object-inspect": "^1.13.3"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/side-channel-weakmap": {++ "version": "1.0.2",++ "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",++ "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",++ "license": "MIT",++ "dependencies": {++ "call-bound": "^1.0.2",++ "es-errors": "^1.3.0",++ "get-intrinsic": "^1.2.5",++ "object-inspect": "^1.13.3",++ "side-channel-map": "^1.0.1"++ },++ "engines": {++ "node": ">= 0.4"++ },++ "funding": {++ "url": "https://github.com/sponsors/ljharb"++ }++ },++ "node_modules/simple-concat": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",++ "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "MIT"++ },++ "node_modules/simple-get": {++ "version": "4.0.1",++ "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",++ "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/feross"++ },++ {++ "type": "patreon",++ "url": "https://www.patreon.com/feross"++ },++ {++ "type": "consulting",++ "url": "https://feross.org/support"++ }++ ],++ "license": "MIT",++ "dependencies": {++ "decompress-response": "^6.0.0",++ "once": "^1.3.1",++ "simple-concat": "^1.0.0"++ }++ },++ "node_modules/statuses": {++ "version": "2.0.2",++ "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",++ "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/string_decoder": {++ "version": "1.3.0",++ "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",++ "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",++ "license": "MIT",++ "dependencies": {++ "safe-buffer": "~5.2.0"++ }++ },++ "node_modules/strip-json-comments": {++ "version": "2.0.1",++ "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",++ "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/strnum": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/strnum/-/strnum-1.1.2.tgz",++ "integrity": "sha512-vrN+B7DBIoTTZjnPNewwhx6cBA/H+IS7rfW68n7XxC1y7uoiGQBxaKzqucGUgavX15dJgiGztLJ8vxuEzwqBdA==",++ "funding": [++ {++ "type": "github",++ "url": "https://github.com/sponsors/NaturalIntelligence"++ }++ ],++ "license": "MIT"++ },++ "node_modules/tar-fs": {++ "version": "2.1.5",++ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",++ "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",++ "license": "MIT",++ "dependencies": {++ "chownr": "^1.1.1",++ "mkdirp-classic": "^0.5.2",++ "pump": "^3.0.0",++ "tar-stream": "^2.1.4"++ }++ },++ "node_modules/tar-stream": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",++ "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",++ "license": "MIT",++ "dependencies": {++ "bl": "^4.0.3",++ "end-of-stream": "^1.4.1",++ "fs-constants": "^1.0.0",++ "inherits": "^2.0.3",++ "readable-stream": "^3.1.1"++ },++ "engines": {++ "node": ">=6"++ }++ },++ "node_modules/toidentifier": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",++ "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",++ "license": "MIT",++ "engines": {++ "node": ">=0.6"++ }++ },++ "node_modules/tunnel-agent": {++ "version": "0.6.0",++ "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",++ "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",++ "license": "Apache-2.0",++ "dependencies": {++ "safe-buffer": "^5.0.1"++ },++ "engines": {++ "node": "*"++ }++ },++ "node_modules/type-is": {++ "version": "1.6.18",++ "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",++ "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",++ "license": "MIT",++ "dependencies": {++ "media-typer": "0.3.0",++ "mime-types": "~2.1.24"++ },++ "engines": {++ "node": ">= 0.6"++ }++ },++ "node_modules/unpipe": {++ "version": "1.0.0",++ "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",++ "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/util-deprecate": {++ "version": "1.0.2",++ "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",++ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",++ "license": "MIT"++ },++ "node_modules/utils-merge": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",++ "integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.4.0"++ }++ },++ "node_modules/vary": {++ "version": "1.1.2",++ "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",++ "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",++ "license": "MIT",++ "engines": {++ "node": ">= 0.8"++ }++ },++ "node_modules/wrappy": {++ "version": "1.0.2",++ "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",++ "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",++ "license": "ISC"++ }++ }++}nnew file mode 100644@@@ -0,0 +1,39 @@++{++ "name": "openvibe-sources",++ "version": "0.1.0",++ "private": true,++ "description": "OpenVibe.Sources: the source registry and the ingestion workers (RSS/Atom, sitemaps, JSON-LD, official APIs) that record provenance, retrieval time, content hashes and explicit failure states, never invented content.",++ "main": "server/index.js",++ "scripts": {++ "start": "node server/index.js",++ "dev": "NODE_ENV=development node server/index.js",++ "test": "node test/run.js",++ "seed": "node scripts/seed.js"++ },++ "keywords": [++ "openvibe",++ "sources",++ "rss",++ "atom",++ "sitemap",++ "json-ld",++ "ingestion"++ ],++ "author": "OpenVibers",++ "license": "AGPL-3.0-only",++ "repository": "github:OpenVibers/OpenVibe.Sources",++ "dependencies": {++ "better-sqlite3": "^11.10.0",++ "dotenv": "^16.4.7",++ "express": "^4.21.2",++ "fast-xml-parser": "^4.5.3",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0"++ },++ "engines": {++ "node": ">=22.0.0"++ },++ "devDependencies": {++ "ajv": "^8.20.0",++ "ajv-formats": "^3.0.1"++ }++}nnew file mode 100644@@@ -0,0 +1,38 @@++#!/usr/bin/env node++'use strict';++/**++ * Load seeds/sources.json into the registry: creates sources that do not exist yet, DISABLED.++ * Never overwrites or enables an existing source. Enable one deliberately after verifying its terms:++ * PATCH /api/v1/sources/:key { "enabled": true } (sources.source.manage).++ *++ * node scripts/seed.js [--db ./data/sources.db]++ */++require('dotenv').config();++const path = require('path');++const { load } = require('../server/config');++const { openDb } = require('../server/db');++const { createRegistry } = require('../server/registry');++++function seed(db, { now = () => Date.now(), by = 'seed' } = {}) {++ const seeds = require(path.join(__dirname, '..', 'seeds', 'sources.json')).sources;++ const registry = createRegistry({ db, now });++ const out = { created: [], kept: [] };++ for (const s of seeds) {++ if (registry.get(s.key)) { out.kept.push(s.key); continue; }++ registry.create({ ...s, enabled: false }, by);++ out.created.push(s.key);++ }++ return out;++}++++if (require.main === module) {++ const args = process.argv.slice(2);++ const i = args.indexOf('--db');++ const config = load();++ const db = openDb(i >= 0 ? args[i + 1] : config.dbPath);++ const r = seed(db);++ console.log(`seeded ${r.created.length} source(s)${r.created.length ? `: ${r.created.join(', ')}` : ''}; kept ${r.kept.length} existing. All seeds are disabled.`);++ db.close();++}++++module.exports = { seed };nnew file mode 100644@@@ -0,0 +1,113 @@++{++ "_about": "One real source per category, all DISABLED. Robots notes were checked by hand on 2026-09-22; terms must be re-verified by a person before a source is enabled (PATCH enabled: true). Seeding never overwrites or enables an existing source.",++ "sources": [++ {++ "key": "nasa-news-releases",++ "name": "NASA news releases",++ "type": "rss",++ "category": "news",++ "homepage_url": "https://www.nasa.gov/",++ "endpoints": ["https://www.nasa.gov/news-release/feed/"],++ "robots_note": "robots.txt checked 2026-09-22: User-agent * Allow /. The fetcher re-checks it on every run.",++ "terms_note": "NASA's media usage guidelines say NASA material is generally not subject to copyright in the US; logos, insignia and third-party material are excluded. Store titles, links and short summaries; link back. Re-verify before enabling.",++ "license_note": "US federal government work in general; per-item credits may restrict reuse.",++ "min_interval_ms": 60000,++ "poll_interval_sec": 1800,++ "max_items": 100,++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"++ },++ {++ "key": "nodejs-blog",++ "name": "Node.js blog",++ "type": "rss",++ "category": "blog",++ "homepage_url": "https://nodejs.org/en/blog",++ "endpoints": ["https://nodejs.org/en/feed/blog.xml"],++ "robots_note": "robots.txt checked 2026-09-22: only /dist/ and /docs/ are disallowed; the feed path is allowed.",++ "terms_note": "Content of the OpenJS Foundation and contributors; read the site's terms before enabling. Titles, links and short summaries only.",++ "min_interval_ms": 60000,++ "poll_interval_sec": 3600,++ "max_items": 50,++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"++ },++ {++ "key": "steam-reviews-portal-2",++ "name": "Steam user review signals: Portal 2 (app 620)",++ "type": "api",++ "category": "reviews",++ "homepage_url": "https://store.steampowered.com/app/620/",++ "endpoints": [{++ "url": "https://store.steampowered.com/appreviews/620?json=1&filter=recent&language=english&purchase_type=all&num_per_page=20",++ "format": "json",++ "items_path": "reviews",++ "item_kind": "review_signal",++ "fields": { "identity": "recommendationid", "published_at_epoch": "timestamp_created", "updated_at_epoch": "timestamp_updated" },++ "extra": {++ "voted_up": "voted_up", "votes_up": "votes_up", "votes_funny": "votes_funny", "weighted_vote_score": "weighted_vote_score",++ "playtime_at_review_min": "author.playtime_at_review", "language": "language", "steam_purchase": "steam_purchase",++ "received_for_free": "received_for_free", "written_during_early_access": "written_during_early_access"++ }++ }],++ "robots_note": "robots.txt checked 2026-09-22: /appreviews is not disallowed.",++ "terms_note": "Documented store endpoint (partner.steamgames.com/doc/store/getreviews); Steam's terms apply. Review text belongs to its authors and is deliberately not mapped: signals only (recommendation, votes, playtime, dates). Verify terms before enabling.",++ "min_interval_ms": 60000,++ "poll_interval_sec": 21600,++ "max_items": 20,++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"++ },++ {++ "key": "dealnews-daily",++ "name": "DealNews daily deals",++ "type": "rss",++ "category": "deals",++ "homepage_url": "https://www.dealnews.com/",++ "endpoints": ["https://www.dealnews.com/?rss=1"],++ "robots_note": "robots.txt checked 2026-09-22: Crawl-delay 2 (honoured); the feed URL is not disallowed.",++ "terms_note": "Copyright DealNews, all rights reserved (feed <copyright>). Headlines, prices exactly as stated and links only; a price is an observation at retrieval time, never a current price. Verify terms before enabling.",++ "min_interval_ms": 60000,++ "poll_interval_sec": 3600,++ "max_items": 100,++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"++ },++ {++ "key": "staff-coupon-codes",++ "name": "Coupon codes published by merchants, entered by staff",++ "type": "manual",++ "category": "coupons",++ "terms_note": "Only codes a merchant published itself, entered with the URL of that page as evidence. Validity and expiry stay unknown unless the merchant states them; no code is 'working' because it was entered.",++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"++ },++ {++ "key": "sec-xbrl-filings",++ "name": "SEC EDGAR: latest XBRL filings",++ "type": "rss",++ "category": "trade",++ "homepage_url": "https://www.sec.gov/structureddata/rss-feeds",++ "endpoints": ["https://www.sec.gov/Archives/edgar/xbrlrss.all.xml"],++ "robots_note": "robots.txt checked 2026-09-22: /Archives/edgar is allowed; /cgi-bin is disallowed, so the EDGAR 'getcurrent' Atom feed is deliberately not used.",++ "terms_note": "Public EDGAR data. SEC fair-access policy: a User-Agent that names the requester and a contact (ours does) and at most 10 requests per second. Informational context only — no advice, no execution.",++ "min_interval_ms": 10000,++ "poll_interval_sec": 900,++ "max_items": 200,++ "enabled": false,++ "review_required": true,++ "sensitivity": "financial",++ "default_indexability": "noindex"++ }++ ]++}nnew file mode 100644@@@ -0,0 +1,94 @@++'use strict';++/**++ * Official APIs (JSON or XML) described by a mapping in the source registry — no code per++ * provider. Each endpoint says where the records are and which field is which:++ *++ * { "url": "https://…", "format": "json", "item_kind": "offer", "items_path": "deals",++ * "fields": { "identity": "dealID", "url": "link", "title": "title", "summary": "…",++ * "author": "…", "published_at": "…", "published_at_epoch": "releaseDate",++ * "updated_at": "…", "updated_at_epoch": "lastChange" },++ * "extra": { "sale_price": "salePrice", "normal_price": "normalPrice", "store_id": "storeID" } }++ *++ * Paths are dot-separated keys (array indexes allowed); `items_path` "" means the body itself is++ * the array. `extra` values are copied as the API gave them (scalars only, strings capped):++ * a missing field is null, never a default. A record without an identity is skipped.++ */++const { parseXml, toText, toIsoDate, epochSecondsToIso, canonicalUrl, toId, ParseError } = require('./util');++++const PARSER_VERSION = 'api@1';++const PATH_RE = /^[A-Za-z0-9_@:$-]+(\.[A-Za-z0-9_@:$-]+)*$/;++++function at(obj, path) {++ if (!path) return obj;++ let cur = obj;++ for (const seg of path.split('.')) {++ if (cur == null || typeof cur !== 'object') return undefined;++ cur = cur[seg];++ }++ return cur;++}++++function scalar(v) {++ if (v == null) return null;++ if (typeof v === 'number' || typeof v === 'boolean') return Number.isFinite(v) || typeof v === 'boolean' ? v : null;++ if (typeof v === 'string') return v.length > 500 ? `${v.slice(0, 499)}…` : v;++ if (typeof v === 'object' && v['#text'] != null) return scalar(v['#text']);++ return null;++}++++/** Validate an endpoint mapping (registry side). Returns an error string or null. */++function checkMapping(ep) {++ if (!['json', 'xml'].includes(ep.format || 'json')) return 'format must be json or xml';++ if (ep.items_path !== undefined && ep.items_path !== '' && !PATH_RE.test(ep.items_path)) return 'items_path is malformed';++ const fields = ep.fields || {};++ if (!fields.identity && !fields.url) return 'fields.identity or fields.url is required';++ for (const [k, p] of Object.entries(fields)) {++ if (!['identity', 'url', 'title', 'summary', 'author', 'published_at', 'published_at_epoch', 'updated_at', 'updated_at_epoch'].includes(k)) return `unknown field ${k}`;++ if (typeof p !== 'string' || !PATH_RE.test(p)) return `fields.${k} path is malformed`;++ }++ const extra = ep.extra || {};++ if (Object.keys(extra).length > 30) return 'at most 30 extra fields';++ for (const [k, p] of Object.entries(extra)) {++ if (!/^[a-z][a-z0-9_]{0,39}$/.test(k) || typeof p !== 'string' || !PATH_RE.test(p)) return `extra.${k} is malformed`;++ }++ if (ep.item_kind !== undefined && !/^[a-z][a-z0-9_]{1,39}$/.test(ep.item_kind)) return 'item_kind is malformed';++ return null;++}++++function parse(text, { url, endpoint = {}, summaryMax = 1000, maxItems = 500 } = {}) {++ const format = endpoint.format || 'json';++ let body;++ if (format === 'xml') body = parseXml(text);++ else {++ try { body = JSON.parse(text); } catch { throw new ParseError('body is not valid JSON'); }++ }++ const list = at(body, endpoint.items_path || '');++ if (!Array.isArray(list)) throw new ParseError(`items_path "${endpoint.items_path || ''}" is not an array in the response`);++ const f = endpoint.fields || {};++ const items = [];++ let skipped = 0;++ const seen = new Set();++ for (const rec of list) {++ if (items.length >= maxItems || !rec || typeof rec !== 'object') { skipped++; continue; }++ const link = f.url ? canonicalUrl(scalar(at(rec, f.url)), url) : null;++ const identity = (f.identity ? toId(scalar(at(rec, f.identity))) : null) || link;++ if (!identity || seen.has(identity)) { skipped++; continue; }++ seen.add(identity);++ const extra = {};++ for (const [k, p] of Object.entries(endpoint.extra || {})) extra[k] = scalar(at(rec, p));++ items.push({++ identity: String(identity),++ kind: endpoint.item_kind || 'record',++ canonical_url: link,++ title: f.title ? toText(scalar(at(rec, f.title)), 500) : null,++ summary: f.summary ? toText(scalar(at(rec, f.summary)), summaryMax) : null,++ authors: f.author ? [toText(scalar(at(rec, f.author)), 200)].filter(Boolean) : [],++ published_at: f.published_at ? toIsoDate(scalar(at(rec, f.published_at))) : f.published_at_epoch ? epochSecondsToIso(scalar(at(rec, f.published_at_epoch))) : null,++ updated_at: f.updated_at ? toIsoDate(scalar(at(rec, f.updated_at))) : f.updated_at_epoch ? epochSecondsToIso(scalar(at(rec, f.updated_at_epoch))) : null,++ fields: extra,++ });++ }++ return { items, skipped, meta: { format } };++}++++module.exports = { parse, PARSER_VERSION, checkMapping };nnew file mode 100644@@@ -0,0 +1,121 @@++'use strict';++/**++ * RSS 2.0, RSS 1.0 (RDF) and Atom 1.0 → items of kind `article`.++ *++ * Identity: the entry's guid/id, else its link. An entry with neither is skipped (counted), never++ * given a made-up identity. Summary is text only (HTML stripped) and capped: Sources keeps++ * metadata and short descriptions, not republishable full text.++ */++const { parseXml, toText, toIsoDate, canonicalUrl, toId, asArray, pick, ParseError } = require('./util');++++const PARSER_VERSION = 'feed@1';++++function atomLink(entry, base) {++ const links = asArray(entry.link);++ const alt = links.find(l => typeof l === 'object' && (!l['@_rel'] || l['@_rel'] === 'alternate') && l['@_href'])++ || links.find(l => typeof l === 'object' && l['@_href'])++ || links.find(l => typeof l === 'string');++ if (!alt) return null;++ return canonicalUrl(typeof alt === 'string' ? alt : alt['@_href'], base);++}++++function authorsOf(list) {++ return asArray(list).map((a) => {++ if (a == null) return null;++ if (typeof a === 'object') return toText(a.name ?? a['#text'] ?? a.__cdata, 200);++ return toText(a, 200);++ }).filter(Boolean).slice(0, 20);++}++++function categoriesOf(list) {++ return asArray(list).map((c) => {++ if (c == null) return null;++ if (typeof c === 'object') return toText(c['@_term'] ?? c['#text'] ?? c.__cdata, 100);++ return toText(c, 100);++ }).filter(Boolean).slice(0, 30);++}++++function rssItem(it, base, summaryMax) {++ const guidNode = Array.isArray(it.guid) ? it.guid[0] : it.guid;++ const guid = toId(guidNode);++ const link = canonicalUrl(pick(it, 'link'), base);++ // RSS: a guid is a permalink unless isPermaLink="false".++ const guidIsPermalink = !(guidNode && typeof guidNode === 'object' && guidNode['@_isPermaLink'] === 'false');++ const url = link || (guid && guidIsPermalink ? canonicalUrl(guid) : null);++ const identity = guid || url;++ if (!identity) return null;++ const enclosure = asArray(it.enclosure).find(e => e && typeof e === 'object' && e['@_url']);++ return {++ identity,++ kind: 'article',++ canonical_url: url,++ title: toText(pick(it, 'title'), 500),++ summary: toText(pick(it, 'description', 'content:encoded'), summaryMax),++ authors: authorsOf(it['dc:creator'] ?? it.author),++ published_at: toIsoDate(pick(it, 'pubDate', 'dc:date')),++ updated_at: toIsoDate(pick(it, 'atom:updated', 'dc:modified')),++ fields: {++ categories: categoriesOf(it.category ?? it['dc:subject']),++ enclosure: enclosure ? { url: canonicalUrl(enclosure['@_url'], base), type: toId(enclosure['@_type'], 100), length: /^\d{1,15}$/.test(String(enclosure['@_length'] || '')) ? Number(enclosure['@_length']) : null } : null,++ comments_url: canonicalUrl(pick(it, 'comments'), base),++ },++ };++}++++function atomEntry(e, base, summaryMax) {++ const id = toId(pick(e, 'id'));++ const url = atomLink(e, base);++ const identity = id || url;++ if (!identity) return null;++ return {++ identity,++ kind: 'article',++ canonical_url: url,++ title: toText(pick(e, 'title'), 500),++ summary: toText(pick(e, 'summary', 'content'), summaryMax),++ authors: authorsOf(e.author),++ published_at: toIsoDate(pick(e, 'published', 'issued')),++ updated_at: toIsoDate(pick(e, 'updated', 'modified')),++ fields: { categories: categoriesOf(e.category) },++ };++}++++/** parse(text, { url, summaryMax, maxItems }) → { items, skipped, meta } or throws ParseError. */++function parse(text, { url, summaryMax = 1000, maxItems = 500 } = {}) {++ const doc = parseXml(text);++ let raw = [];++ let meta = {};++ let make;++ if (doc.rss && doc.rss.channel) {++ const ch = Array.isArray(doc.rss.channel) ? doc.rss.channel[0] : doc.rss.channel;++ raw = asArray(ch.item);++ meta = { format: 'rss2', title: toText(pick(ch, 'title'), 300), link: canonicalUrl(pick(ch, 'link'), url) };++ make = (it) => rssItem(it, url, summaryMax);++ } else if (doc['rdf:RDF']) {++ const rdf = doc['rdf:RDF'];++ raw = asArray(rdf.item);++ const ch = Array.isArray(rdf.channel) ? rdf.channel[0] : rdf.channel;++ meta = { format: 'rss1', title: ch ? toText(pick(ch, 'title'), 300) : null };++ make = (it) => rssItem({ ...it, guid: it['@_rdf:about'] ? { '#text': it['@_rdf:about'] } : undefined }, url, summaryMax);++ } else if (doc.feed) {++ const feed = doc.feed;++ raw = asArray(feed.entry);++ meta = { format: 'atom', title: toText(pick(feed, 'title'), 300), link: atomLink(feed, url) };++ make = (e) => atomEntry(e, url, summaryMax);++ } else {++ throw new ParseError('not an RSS or Atom document');++ }++ const items = [];++ let skipped = 0;++ const seen = new Set();++ for (const r of raw) {++ if (items.length >= maxItems) { skipped++; continue; }++ const it = r && typeof r === 'object' ? make(r) : null;++ if (!it || seen.has(it.identity)) { skipped++; continue; }++ seen.add(it.identity);++ items.push(it);++ }++ return { items, skipped, meta };++}++++module.exports = { parse, PARSER_VERSION };nnew file mode 100644@@@ -0,0 +1,21 @@++'use strict';++/**++ * Adapter registry: source type → parser. `manual` sources have no fetcher; their items are++ * entered through the API with the evidence URL they came from.++ */++const feed = require('./feed');++const sitemap = require('./sitemap');++const jsonld = require('./jsonld');++const api = require('./api');++++const ADAPTERS = {++ rss: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },++ atom: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/atom+xml, application/rss+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },++ sitemap: { parse: sitemap.parse, version: sitemap.PARSER_VERSION, accept: 'application/xml, text/xml;q=0.9, */*;q=0.1' },++ jsonld: { parse: jsonld.parse, version: jsonld.PARSER_VERSION, accept: 'text/html, application/xhtml+xml;q=0.9, */*;q=0.1' },++ api: { parse: api.parse, version: api.PARSER_VERSION, accept: 'application/json, application/xml;q=0.9, */*;q=0.1' },++};++++const MANUAL_VERSION = 'manual@1';++++module.exports = { ADAPTERS, MANUAL_VERSION, checkApiMapping: api.checkMapping };nnew file mode 100644@@@ -0,0 +1,193 @@++'use strict';++/**++ * JSON-LD (schema.org) embedded in HTML pages → items of kind product | offer | review | article.++ *++ * Types read: Product, Offer, AggregateOffer, Review, NewsArticle, BlogPosting (and Article and++ * its news subtypes). Values are copied as the page states them: prices stay the strings the page++ * gave, ratings are the page's own numbers, availability is the schema.org term. Nothing is++ * computed, averaged or defaulted; a missing price is null. Embedded reviews keep author, rating++ * and date only (review text belongs to its author).++ *++ * A page without any readable JSON-LD block is a parse error: the source promised structured data++ * and did not deliver it, and the run must say so instead of producing nothing silently.++ */++const crypto = require('crypto');++const { toText, toIsoDate, canonicalUrl, toId, asArray, ParseError } = require('./util');++++const PARSER_VERSION = 'jsonld@1';++const ARTICLE_TYPES = new Set(['NewsArticle', 'BlogPosting', 'Article', 'ReportageNewsArticle', 'AnalysisNewsArticle', 'OpinionNewsArticle', 'BackgroundNewsArticle']);++const KIND = { Product: 'product', Offer: 'offer', AggregateOffer: 'offer', Review: 'review' };++++function typesOf(node) {++ return asArray(node && node['@type']).map(t => String(t).replace(/^https?:\/\/schema\.org\//, ''));++}++++function kindOf(node) {++ for (const t of typesOf(node)) {++ if (KIND[t]) return { kind: KIND[t], type: t };++ if (ARTICLE_TYPES.has(t)) return { kind: 'article', type: t };++ }++ return null;++}++++function extractBlocks(html) {++ const blocks = [];++ const re = /<script\b[^>]*type\s*=\s*["']?application\/ld\+json[^>]*>([\s\S]*?)<\/script\s*>/gi;++ let m;++ while ((m = re.exec(html)) && blocks.length < 50) blocks.push(m[1]);++ return blocks;++}++++function flatten(value, out, depth = 0) {++ if (depth > 3 || value == null) return;++ if (Array.isArray(value)) { for (const v of value) flatten(v, out, depth + 1); return; }++ if (typeof value !== 'object') return;++ if (value['@graph']) flatten(value['@graph'], out, depth + 1);++ if (value['@type']) out.push(value);++}++++const str = (v, max = 200) => (v == null || typeof v === 'object' ? null : toText(String(v), max));++const nameOf = (v) => (v == null ? null : typeof v === 'object' ? str(Array.isArray(v) ? (v[0] && v[0].name) : v.name) : str(v));++/** A number the page stated as a number or numeric string; anything else → null. */++const numeric = (v) => {++ if (typeof v === 'number' && Number.isFinite(v)) return v;++ if (typeof v === 'string' && /^-?\d+(\.\d+)?$/.test(v.trim())) return Number(v.trim());++ return null;++};++/** A price exactly as written, if it is a plain decimal; else null. */++const priceOf = (v) => {++ if (typeof v === 'number' && Number.isFinite(v)) return String(v);++ if (typeof v === 'string' && /^\d+(\.\d+)?$/.test(v.trim())) return v.trim();++ return null;++};++const term = (v) => (typeof v === 'string' ? v.replace(/^https?:\/\/schema\.org\//, '').slice(0, 60) : null);++const imageOf = (v, base) => {++ const first = asArray(v)[0];++ return canonicalUrl(first && typeof first === 'object' ? first.url || first.contentUrl : first, base);++};++++function offerFields(o, base) {++ if (!o || typeof o !== 'object') return null;++ return {++ type: typesOf(o)[0] || null,++ price: priceOf(o.price),++ low_price: priceOf(o.lowPrice),++ high_price: priceOf(o.highPrice),++ offer_count: numeric(o.offerCount),++ currency: typeof o.priceCurrency === 'string' && /^[A-Z]{3}$/.test(o.priceCurrency) ? o.priceCurrency : null,++ availability: term(o.availability),++ condition: term(o.itemCondition),++ valid_until: toIsoDate(o.priceValidUntil),++ url: canonicalUrl(o.url, base),++ seller: nameOf(o.seller),++ };++}++++function ratingFields(r) {++ if (!r || typeof r !== 'object') return null;++ const out = {++ value: numeric(r.ratingValue), best: numeric(r.bestRating), worst: numeric(r.worstRating),++ count: numeric(r.ratingCount), review_count: numeric(r.reviewCount),++ };++ return out.value == null && out.count == null && out.review_count == null ? null : out;++}++++function identityFor(node, url, pageUrl, type, ...keys) {++ const id = typeof node['@id'] === 'string' ? canonicalUrl(node['@id'], pageUrl) || toId(node['@id']) : null;++ if (id) return id;++ if (url && url !== pageUrl) return url;++ const key = keys.find(k => k);++ if (key) return `${pageUrl}#${type}:${crypto.createHash('sha256').update(String(key)).digest('hex').slice(0, 16)}`;++ return null;++}++++function toItem(node, pageUrl, summaryMax) {++ const k = kindOf(node);++ if (!k) return null;++ const url = canonicalUrl(node.url, pageUrl);++ if (k.kind === 'product') {++ const sku = toId(node.sku, 100);++ const gtin = toId(node.gtin13 || node.gtin12 || node.gtin14 || node.gtin8 || node.gtin, 20);++ const name = str(node.name, 500);++ const identity = identityFor(node, url, pageUrl, 'Product', sku, gtin, name);++ if (!identity) return null;++ return {++ identity, kind: 'product', canonical_url: url || pageUrl,++ title: name, summary: toText(node.description, summaryMax), authors: [],++ published_at: null, updated_at: null,++ fields: {++ schema_type: k.type, sku, gtin, mpn: toId(node.mpn, 100), brand: nameOf(node.brand),++ image: imageOf(node.image, pageUrl),++ offers: asArray(node.offers).map(o => offerFields(o, pageUrl)).filter(Boolean).slice(0, 20),++ aggregate_rating: ratingFields(node.aggregateRating),++ reviews: asArray(node.review).slice(0, 10).map(r => (r && typeof r === 'object' ? {++ author: nameOf(r.author), rating: ratingFields(r.reviewRating), date: toIsoDate(r.datePublished),++ } : null)).filter(Boolean),++ },++ };++ }++ if (k.kind === 'offer') {++ const f = offerFields(node, pageUrl);++ const itemName = nameOf(node.itemOffered) || str(node.name, 500);++ const identity = identityFor(node, url, pageUrl, k.type, toId(node.sku, 100), itemName && `${itemName}|${f.price}|${f.currency}`);++ if (!identity) return null;++ return {++ identity, kind: 'offer', canonical_url: url || pageUrl, title: itemName, summary: toText(node.description, summaryMax),++ authors: [], published_at: null, updated_at: null, fields: { schema_type: k.type, ...f },++ };++ }++ if (k.kind === 'review') {++ const title = str(node.name, 500) || (nameOf(node.itemReviewed) ? `Review: ${nameOf(node.itemReviewed)}` : null);++ const authors = asArray(node.author).map(nameOf).filter(Boolean).slice(0, 10);++ const identity = identityFor(node, url, pageUrl, 'Review', `${nameOf(node.itemReviewed)}|${authors.join(',')}|${node.datePublished || ''}`);++ if (!identity) return null;++ return {++ identity, kind: 'review', canonical_url: url || pageUrl, title,++ summary: toText(node.reviewBody || node.description, summaryMax), authors,++ published_at: toIsoDate(node.datePublished), updated_at: toIsoDate(node.dateModified),++ fields: {++ schema_type: 'Review', item_reviewed: nameOf(node.itemReviewed),++ item_reviewed_type: node.itemReviewed && typeof node.itemReviewed === 'object' ? typesOf(node.itemReviewed)[0] || null : null,++ rating: ratingFields(node.reviewRating), publisher: nameOf(node.publisher),++ },++ };++ }++ // article++ const headline = str(node.headline || node.name, 500);++ const identity = identityFor(node, url, pageUrl, k.type, headline);++ if (!identity) return null;++ return {++ identity, kind: 'article', canonical_url: url || pageUrl, title: headline,++ summary: toText(node.description, summaryMax),++ authors: asArray(node.author).map(nameOf).filter(Boolean).slice(0, 20),++ published_at: toIsoDate(node.datePublished), updated_at: toIsoDate(node.dateModified),++ fields: {++ schema_type: k.type, publisher: nameOf(node.publisher), section: str(node.articleSection, 100),++ image: imageOf(node.image, pageUrl), language: str(node.inLanguage, 20),++ },++ };++}++++function parse(text, { url, summaryMax = 1000, maxItems = 500 } = {}) {++ const blocks = extractBlocks(text);++ if (!blocks.length) throw new ParseError('no application/ld+json block on the page');++ const nodes = [];++ let bad = 0;++ for (const b of blocks) {++ try { flatten(JSON.parse(b.trim()), nodes); } catch { bad++; }++ }++ if (bad === blocks.length) throw new ParseError('no JSON-LD block is valid JSON');++ const items = [];++ let skipped = bad;++ const seen = new Set();++ for (const n of nodes) {++ if (!kindOf(n)) continue; // Organization, BreadcrumbList… are not items++ const it = toItem(n, url, summaryMax);++ if (!it || seen.has(it.identity) || items.length >= maxItems) { skipped++; continue; }++ seen.add(it.identity);++ items.push(it);++ }++ return { items, skipped, meta: { format: 'jsonld', blocks: blocks.length } };++}++++module.exports = { parse, PARSER_VERSION, extractBlocks };nnew file mode 100644@@@ -0,0 +1,60 @@++'use strict';++/**++ * Sitemaps (sitemaps.org 0.9): a <urlset> → items of kind `url` (loc, lastmod, plus news:news++ * title/date when the Google News extension is present); a <sitemapindex> → items of kind++ * `sitemap` (child sitemaps are listed, not followed: register a child as its own source to++ * ingest it, so every request stays visible and rate-limited).++ * Sitemap entries carry no body; only what the file states is stored.++ */++const { parseXml, toText, toIsoDate, canonicalUrl, asArray, pick, ParseError } = require('./util');++++const PARSER_VERSION = 'sitemap@1';++++function parse(text, { url, maxItems = 500 } = {}) {++ const doc = parseXml(text);++ const items = [];++ let skipped = 0;++ const seen = new Set();++ const push = (it) => {++ if (!it || seen.has(it.identity) || items.length >= maxItems) { skipped++; return; }++ seen.add(it.identity);++ items.push(it);++ };++ if (doc.urlset) {++ for (const u of asArray(doc.urlset.url)) {++ const loc = u && typeof u === 'object' ? canonicalUrl(pick(u, 'loc')) : null;++ if (!loc) { skipped++; continue; }++ const news = u['news:news'] && (Array.isArray(u['news:news']) ? u['news:news'][0] : u['news:news']);++ push({++ identity: loc,++ kind: 'url',++ canonical_url: loc,++ title: news ? toText(pick(news, 'news:title'), 500) : null,++ summary: null,++ authors: [],++ published_at: news ? toIsoDate(pick(news, 'news:publication_date')) : null,++ updated_at: toIsoDate(pick(u, 'lastmod')),++ fields: {++ changefreq: toText(pick(u, 'changefreq'), 20),++ priority: /^(0(\.\d+)?|1(\.0+)?)$/.test(String(pick(u, 'priority') ?? '')) ? Number(pick(u, 'priority')) : null,++ publication: news && news['news:publication'] ? toText(pick(news['news:publication'], 'news:name'), 200) : null,++ },++ });++ }++ return { items, skipped, meta: { format: 'urlset' } };++ }++ if (doc.sitemapindex) {++ for (const s of asArray(doc.sitemapindex.sitemap)) {++ const loc = s && typeof s === 'object' ? canonicalUrl(pick(s, 'loc')) : null;++ if (!loc) { skipped++; continue; }++ push({++ identity: loc, kind: 'sitemap', canonical_url: loc, title: null, summary: null, authors: [],++ published_at: null, updated_at: toIsoDate(pick(s, 'lastmod')), fields: {},++ });++ }++ return { items, skipped, meta: { format: 'sitemapindex', base: url } };++ }++ throw new ParseError('not a sitemap (<urlset> or <sitemapindex>)');++}++++module.exports = { parse, PARSER_VERSION };nnew file mode 100644@@@ -0,0 +1,167 @@++'use strict';++/**++ * Helpers every adapter shares. Adapters turn a fetched body into parsed items and nothing++ * else: a field the source did not provide stays null — never guessed, defaulted or invented.++ */++const crypto = require('crypto');++const { XMLParser } = require('fast-xml-parser');++++class ParseError extends Error {++ constructor(detail) { super(detail); this.code = 'parse_error'; }++}++++function sha256(buf) {++ return crypto.createHash('sha256').update(buf).digest('hex');++}++++function canonicalJson(value) {++ if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`;++ if (value && typeof value === 'object') {++ return `{${Object.keys(value).sort().filter(k => value[k] !== undefined).map(k => `${JSON.stringify(k)}:${canonicalJson(value[k])}`).join(',')}}`;++ }++ return JSON.stringify(value === undefined ? null : value);++}++++/** Decode a body: charset from Content-Type, else the XML declaration / meta tag, else UTF-8. */++function decodeBody(buf, contentType) {++ let charset = /charset=["']?([\w-]+)/i.exec(String(contentType || ''))?.[1];++ if (!charset) {++ const head = buf.subarray(0, 1024).toString('latin1');++ charset = /<\?xml[^>]*encoding=["']([\w-]+)["']/i.exec(head)?.[1] || /<meta[^>]+charset=["']?([\w-]+)/i.exec(head)?.[1];++ }++ let label = String(charset || 'utf-8').toLowerCase();++ if (label === 'iso-8859-1' || label === 'latin1') label = 'windows-1252';++ try {++ return new TextDecoder(label, { fatal: false }).decode(buf).replace(/^\uFEFF/, '');++ } catch {++ return new TextDecoder('utf-8').decode(buf).replace(/^\uFEFF/, '');++ }++}++++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', hellip: '…', mdash: '—', ndash: '–', rsquo: '’', lsquo: '‘', rdquo: '”', ldquo: '“' };++++function decodeEntities(s) {++ return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {++ if (e[0] === '#') {++ const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);++ return Number.isFinite(code) && code > 0 && code < 0x110000 ? String.fromCodePoint(code) : '';++ }++ return ENTITIES[e.toLowerCase()] ?? m;++ });++}++++/** HTML or text → plain text, whitespace collapsed, capped. Scripts and styles are dropped whole. */++function toText(value, max = 1000) {++ if (value == null) return null;++ // Bounded input: only a short summary survives anyway, and the regexes below stay linear.++ let s = String(typeof value === 'object' ? (value['#text'] ?? value.__cdata ?? '') : value).slice(0, 100000);++ // XML entities first (the parser leaves them), then markup, then HTML entities of the markup.++ s = decodeEntities(String(s))++ .replace(/<(script|style|noscript)[\s\S]*?<\/\1>/gi, ' ')++ .replace(/<br\s*\/?>|<\/p>|<\/li>|<\/h\d>/gi, ' ')++ .replace(/<[^>]*>/g, ' ');++ s = decodeEntities(s).replace(/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F]/g, '').replace(/\s+/g, ' ').trim();++ if (!s) return null;++ return s.length > max ? `${s.slice(0, max - 1).trimEnd()}…` : s;++}++++/** RFC 822 / RFC 3339 / ISO dates → ISO string; anything unparseable → null (never "now"). */++function toIsoDate(value) {++ if (value == null) return null;++ const s = decodeEntities(String(typeof value === 'object' ? value['#text'] ?? '' : value)).trim();++ if (!s || s.length > 64) return null;++ // Refuse bare numbers: ambiguous epoch units are guesses.++ if (/^\d+$/.test(s)) return null;++ const t = Date.parse(s);++ if (!Number.isFinite(t)) return null;++ const year = new Date(t).getUTCFullYear();++ if (year < 1990 || year > 2200) return null;++ return new Date(t).toISOString();++}++++/** Epoch seconds (for APIs that document them) → ISO, or null. */++function epochSecondsToIso(value) {++ const n = Number(value);++ if (!Number.isFinite(n) || n <= 0) return null;++ const d = new Date(n * 1000);++ const y = d.getUTCFullYear();++ return y >= 1990 && y <= 2200 ? d.toISOString() : null;++}++++const TRACKING = /^(utm_[a-z]+|fbclid|gclid|mc_cid|mc_eid|_hsenc|_hsmi|mkt_tok)$/i;++++/** http(s) URL → canonical form (lowercase host, no fragment, no tracking params, no default port); else null. */++function canonicalUrl(value, base) {++ if (value == null) return null;++ const raw = decodeEntities(String(typeof value === 'object' ? value['@_href'] ?? value['#text'] ?? '' : value)).trim();++ if (!raw || raw.length > 2048) return null;++ let url;++ try { url = base ? new URL(raw, base) : new URL(raw); } catch { return null; }++ if (url.protocol !== 'http:' && url.protocol !== 'https:') return null;++ if (url.username || url.password) return null;++ url.hash = '';++ url.hostname = url.hostname.toLowerCase();++ if ((url.protocol === 'https:' && url.port === '443') || (url.protocol === 'http:' && url.port === '80')) url.port = '';++ for (const k of [...url.searchParams.keys()]) if (TRACKING.test(k)) url.searchParams.delete(k);++ return url.toString();++}++++/** A plain identifier string (guid, sku…) or null. */++function toId(value, max = 512) {++ if (value == null) return null;++ const s = decodeEntities(String(typeof value === 'object' ? value['#text'] ?? '' : value)).trim();++ return s && s.length <= max ? s : null;++}++++function asArray(v) {++ if (v == null) return [];++ return Array.isArray(v) ? v : [v];++}++++/**++ * Parse XML safely: a DOCTYPE with entity declarations is refused (entity expansion is how XML++ * bombs work) and the parser expands no entities at all — the helpers above decode the predefined++ * and numeric ones per value (large real feeds carry thousands of &, which trips the++ * parser's own expansion limit).++ */++function parseXml(text) {++ if (/<!DOCTYPE[^>]*\[/i.test(text) || /<!ENTITY/i.test(text)) throw new ParseError('XML with entity declarations is refused');++ const parser = new XMLParser({++ ignoreAttributes: false,++ attributeNamePrefix: '@_',++ removeNSPrefix: false,++ processEntities: false,++ trimValues: true,++ parseTagValue: false,++ parseAttributeValue: false,++ cdataPropName: '__cdata',++ });++ try {++ return parser.parse(text, true);++ } catch (err) {++ throw new ParseError(`not well-formed XML: ${String(err.message).split('\n')[0].slice(0, 200)}`);++ }++}++++/** First non-empty text value of the given keys on an XML node. */++function pick(node, ...keys) {++ for (const k of keys) {++ const v = node && node[k];++ if (v == null) continue;++ const first = Array.isArray(v) ? v[0] : v;++ if (first == null) continue;++ if (typeof first === 'object') {++ const t = first['#text'] ?? first.__cdata;++ if (t != null && String(t).trim()) return t;++ if (first['@_href']) return first;++ continue;++ }++ if (String(first).trim()) return first;++ }++ return null;++}++++module.exports = {++ ParseError, sha256, canonicalJson, decodeBody, toText, toIsoDate, epochSecondsToIso, canonicalUrl, toId,++ asArray, parseXml, pick, decodeEntities,++};nnew file mode 100644@@@ -0,0 +1,113 @@++'use strict';++/**++ * Items API (service tokens).++ *++ * sources.item.read++ * GET /api/v1/items?source=&category=&after=<change_seq>&limit=&include_removed=1++ * items in change order (created, updated, removed); resume from next_after. Each page++ * carries the staleness of every source it contains, so a consumer never mistakes an old++ * observation for a current one.++ * GET /api/v1/items/:id?revisions=1++ * sources.source.manage++ * POST /api/v1/sources/:key/items add or revise an item of a `manual` source (evidence URL required)++ * DELETE /api/v1/items/:id { reason } — takedown/licence removal, sticky++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { CAPS } = require('../auth');++const { CATEGORIES } = require('../registry');++const { MANUAL_VERSION } = require('../adapters');++const { toText, toIsoDate, canonicalUrl, toId } = require('../adapters/util');++++function manualItem(body) {++ if (!body || typeof body !== 'object' || Array.isArray(body)) return { error: 'body must be an object' };++ const url = canonicalUrl(body.url);++ if (!url) return { error: 'url (the evidence: where this was published) must be an http(s) URL' };++ const title = toText(body.title, 500);++ if (!title) return { error: 'title is required' };++ const kind = body.kind === undefined ? 'record' : body.kind;++ if (!/^[a-z][a-z0-9_]{1,39}$/.test(String(kind))) return { error: 'kind is malformed' };++ if (body.published_at != null && !toIsoDate(body.published_at)) return { error: 'published_at must be a date the source states' };++ const fields = body.fields == null ? {} : body.fields;++ if (typeof fields !== 'object' || Array.isArray(fields) || Object.keys(fields).length > 30) return { error: 'fields must be an object of at most 30 scalars' };++ for (const [k, v] of Object.entries(fields)) {++ if (!/^[a-z][a-z0-9_]{0,39}$/.test(k)) return { error: `fields.${k} is malformed` };++ if (!(v === null || typeof v === 'boolean' || (typeof v === 'number' && Number.isFinite(v)) || (typeof v === 'string' && v.length <= 500))) return { error: `fields.${k} must be a scalar` };++ }++ const identity = body.identity == null ? url : toId(body.identity);++ if (!identity) return { error: 'identity is malformed' };++ return {++ item: {++ identity, kind, canonical_url: url, title, summary: toText(body.summary, 1000),++ authors: Array.isArray(body.authors) ? body.authors.map(a => toText(a, 200)).filter(Boolean).slice(0, 20) : [],++ published_at: body.published_at == null ? null : toIsoDate(body.published_at),++ updated_at: body.updated_at == null ? null : toIsoDate(body.updated_at),++ fields,++ },++ };++}++++function itemsRouter({ db, registry, items, auth, relay, now }) {++ const router = express.Router();++ const read = auth.requireCap(CAPS.items);++ const manage = auth.requireCap(CAPS.manage);++++ router.get('/api/v1/items', read, (req, res) => {++ const source = req.query.source ? String(req.query.source) : null;++ const category = req.query.category ? String(req.query.category) : null;++ if (category && !CATEGORIES.includes(category)) return http.sendProblem(res, 400, 'sources.bad_query', { detail: `category must be one of ${CATEGORIES.join('|')}`, ctx: req.ov });++ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;++ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 100, 1), 500);++ const page = items.list({ source, category, after, limit, includeRemoved: req.query.include_removed === '1' });++ const keys = [...new Set(page.items.map(i => i.source_key))];++ const sources = {};++ for (const k of keys) {++ const row = registry.get(k);++ if (row) {++ const h = registry.health(row);++ sources[k] = { status: h.status, stale: h.stale, last_success_at: h.last_success_at };++ }++ }++ res.json({ ...page, sources });++ });++++ router.get('/api/v1/items/:id', read, (req, res) => {++ const item = items.get(String(req.params.id), { revisions: req.query.revisions === '1' });++ if (!item) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such item', ctx: req.ov });++ const row = registry.get(item.source_key);++ const h = row ? registry.health(row) : null;++ res.json({ item, source: h ? { key: item.source_key, status: h.status, stale: h.stale, last_success_at: h.last_success_at } : null });++ });++++ router.post('/api/v1/sources/:key/items', manage, (req, res) => {++ const row = registry.get(req.params.key);++ if (!row) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such source', ctx: req.ov });++ if (row.type !== 'manual') return http.sendProblem(res, 409, 'sources.not_manual', { detail: 'items of fetched sources come from their fetches only', ctx: req.ov });++ const source = registry.fromRow(row);++ if (!source.enabled) return http.sendProblem(res, 409, 'sources.disabled', { detail: 'the source is disabled', ctx: req.ov });++ const m = manualItem(req.body);++ if (m.error) return http.sendProblem(res, 422, 'sources.bad_item', { detail: m.error, ctx: req.ov });++ const t = now();++ const counts = db.transaction(() => items.ingest(source, [m.item], { runId: null, rawBodyHash: null, parserVersion: MANUAL_VERSION, retrievedAt: t, enteredBy: req.principal.sub }))();++ db.prepare('UPDATE sources SET last_success_at = ?, last_run_at = ? WHERE key = ?').run(t, t, source.key);++ if (relay) relay.flush().catch(() => {});++ const saved = db.prepare('SELECT id FROM items WHERE source_key = ? AND identity = ?').get(source.key, m.item.identity);++ const outcome = counts.created ? 'created' : counts.updated ? 'updated' : counts.skipped ? 'removed' : 'unchanged';++ res.status(counts.created ? 201 : 200).json({ outcome, item: items.get(saved.id) });++ });++++ router.delete('/api/v1/items/:id', manage, (req, res) => {++ const reason = toText(req.body && req.body.reason, 300);++ if (!reason) return http.sendProblem(res, 422, 'sources.reason_required', { detail: 'a removal needs a reason (takedown, licence, error…)', ctx: req.ov });++ const found = items.get(String(req.params.id));++ if (!found) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such item', ctx: req.ov });++ const source = registry.fromRow(registry.get(found.source_key));++ const item = items.remove(source, found.id, reason, req.principal.sub);++ if (relay) relay.flush().catch(() => {});++ res.json({ item });++ });++++ return router;++}++++module.exports = { itemsRouter, manualItem };nnew file mode 100644@@@ -0,0 +1,128 @@++'use strict';++/**++ * Registry and fetch-run API (service tokens, one capability per route).++ *++ * sources.source.read++ * GET /api/v1/sources every source with health and staleness++ * GET /api/v1/sources/:key++ * GET /api/v1/sources/:key/runs?before=&limit= fetch runs, newest first++ * GET /api/v1/runs?state=&source=&after=&limit= fetch runs across sources, oldest first++ * GET /api/v1/health status counts and every source that is not healthy++ * sources.source.manage++ * POST /api/v1/sources create (disabled unless enabled: true + notes)++ * PATCH /api/v1/sources/:key change fields (enable/disable too)++ * DELETE /api/v1/sources/:key only while it has no items++ * POST /api/v1/sources/:key/fetch run now (409 while a run is in flight)++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { CAPS } = require('../auth');++const { RegistryError } = require('../registry');++++const STATES = ['ok', 'not_modified', 'http_error', 'timeout', 'robots_denied', 'parse_error', 'rate_limited', 'disabled'];++++function runView(r) {++ const iso = (v) => (v == null ? null : new Date(v).toISOString());++ return {++ id: r.id, source_key: r.source_key, endpoint_url: r.endpoint_url, trigger: r.trigger,++ started_at: iso(r.started_at), finished_at: iso(r.finished_at), state: r.state, http_status: r.http_status,++ error_code: r.error_code, detail: r.detail, bytes: r.bytes, raw_body_hash: r.raw_body_hash, parser_version: r.parser_version,++ items: { seen: r.items_seen, created: r.items_created, updated: r.items_updated, unchanged: r.items_unchanged, skipped: r.items_skipped },++ seq: r.rid,++ };++}++++function sourcesRouter({ db, registry, ingest, auth }) {++ const router = express.Router();++ const read = auth.requireCap(CAPS.read);++ const manage = auth.requireCap(CAPS.manage);++++ const fail = (res, req, err) => {++ if (err instanceof RegistryError) {++ const status = err.code === 'sources.exists' || err.code === 'sources.has_items' ? 409 : 422;++ return http.sendProblem(res, status, err.code, { detail: err.message, ctx: req.ov });++ }++ throw err;++ };++ const notFound = (res, req) => http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such source', ctx: req.ov });++++ router.get('/api/v1/sources', read, (req, res) => {++ res.json({ sources: registry.all().map(registry.view) });++ });++++ router.get('/api/v1/sources/:key', read, (req, res) => {++ const row = registry.get(req.params.key);++ if (!row) return notFound(res, req);++ res.json({ source: registry.view(row) });++ });++++ router.get('/api/v1/sources/:key/runs', read, (req, res) => {++ if (!registry.get(req.params.key)) return notFound(res, req);++ const before = /^\d{1,15}$/.test(String(req.query.before || '')) ? Number(req.query.before) : Number.MAX_SAFE_INTEGER;++ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 50, 1), 200);++ const rows = db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? AND rid < ? ORDER BY rid DESC LIMIT ?').all(req.params.key, before, limit);++ res.json({ runs: rows.map(runView), next_before: rows.length === limit ? rows[rows.length - 1].rid : null });++ });++++ router.get('/api/v1/runs', read, (req, res) => {++ const state = req.query.state ? String(req.query.state) : null;++ if (state && state !== 'failed' && !STATES.includes(state)) return http.sendProblem(res, 400, 'sources.bad_query', { detail: `state must be failed or one of ${STATES.join('|')}`, ctx: req.ov });++ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;++ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 100, 1), 500);++ const rows = db.prepare(`SELECT * FROM fetch_runs WHERE rid > @after AND (@source IS NULL OR source_key = @source)++ AND (@state IS NULL OR state = @state OR (@state = 'failed' AND state IN ('http_error','timeout','robots_denied','parse_error','rate_limited')))++ ORDER BY rid LIMIT @limit`).all({ after, source: req.query.source ? String(req.query.source) : null, state, limit });++ res.json({ runs: rows.map(runView), next_after: rows.length ? rows[rows.length - 1].rid : after });++ });++++ router.get('/api/v1/health', read, (req, res) => {++ const views = registry.all().map(r => ({ key: r.key, category: r.category, type: r.type, ...registry.health(r) }));++ const counts = {};++ for (const v of views) counts[v.status] = (counts[v.status] || 0) + 1;++ res.json({ counts, attention: views.filter(v => ['failing', 'stale', 'never_fetched'].includes(v.status)) });++ });++++ router.post('/api/v1/sources', manage, (req, res) => {++ try {++ const row = registry.create(req.body, req.principal.sub);++ res.status(201).json({ source: registry.view(row) });++ } catch (err) { fail(res, req, err); }++ });++++ router.patch('/api/v1/sources/:key', manage, (req, res) => {++ try {++ const row = registry.patch(req.params.key, req.body, req.principal.sub);++ if (!row) return notFound(res, req);++ res.json({ source: registry.view(row) });++ } catch (err) { fail(res, req, err); }++ });++++ router.delete('/api/v1/sources/:key', manage, (req, res) => {++ try {++ const ok = db.transaction(() => {++ const r = registry.remove(req.params.key);++ if (r) {++ db.prepare('DELETE FROM fetch_runs WHERE source_key = ?').run(req.params.key);++ db.prepare('DELETE FROM endpoint_state WHERE source_key = ?').run(req.params.key);++ }++ return r;++ })();++ if (!ok) return notFound(res, req);++ res.status(204).end();++ } catch (err) { fail(res, req, err); }++ });++++ router.post('/api/v1/sources/:key/fetch', manage, async (req, res, next) => {++ try {++ const out = await ingest.run(req.params.key, { trigger: 'manual' });++ if (!out) return notFound(res, req);++ if (out.manual) return http.sendProblem(res, 409, 'sources.manual_source', { detail: 'a manual source is not fetched; add items with POST /api/v1/sources/:key/items', ctx: req.ov });++ if (out.busy) return http.sendProblem(res, 409, 'sources.fetch_in_flight', { detail: 'a run of this source is in flight', ctx: req.ov });++ res.json({ runs: out.runs.map(r => runView({ ...r, rid: null })) });++ } catch (err) { next(err); }++ });++++ return router;++}++++module.exports = { sourcesRouter, runView };nnew file mode 100644@@@ -0,0 +1,80 @@++'use strict';++/** Express app: request context, the v1 API, health/readiness. */++const express = require('express');++const { http } = require('openvibe-contracts');++const { sourcesRouter } = require('./api/sources');++const { itemsRouter } = require('./api/items');++const pkg = require('../package.json');++++function createApp({ config, db, registry, items, ingest, scheduler, auth, keys, outbox, relay, now, log = console }) {++ const app = express();++ app.disable('x-powered-by');++ app.set('trust proxy', 'loopback');++ app.use(http.middleware());++ app.use((req, res, next) => {++ res.setHeader('X-Content-Type-Options', 'nosniff');++ res.setHeader('Cache-Control', 'no-store');++ next();++ });++ app.use('/api', express.json({ limit: '256kb', type: ['application/json', 'application/*+json'] }));++++ app.get('/api/health', (_req, res) => {++ res.json({ status: 'ok', service: 'openvibe-sources', version: pkg.version });++ });++++ app.get('/api/ready', (_req, res) => {++ let dbOk = false;++ try { dbOk = db.prepare('SELECT 1 AS ok').get().ok === 1; } catch { dbOk = false; }++ const checks = { db: dbOk, key: keys.loaded(), worker: !config.worker.enabled || scheduler.running() };++ const ready = Object.values(checks).every(Boolean);++ let sources = null;++ if (dbOk) {++ sources = {};++ for (const r of registry.all()) {++ const s = registry.health(r).status;++ sources[s] = (sources[s] || 0) + 1;++ }++ }++ res.status(ready ? 200 : 503).json({++ status: ready ? 'ready' : 'not_ready',++ checks,++ sources,++ runs_in_flight: ingest.inflight().length,++ outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,++ });++ });++++ app.use(sourcesRouter({ db, registry, ingest, auth }));++ app.use(itemsRouter({ db, registry, items, auth, relay, now }));++++ app.get('/', (_req, res) => {++ res.type('text/plain').send([++ 'OpenVibe.Sources: the source registry and ingestion workers for the network\'s publication products.',++ 'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',++ 'recorded as a failure and never replaced with invented content.',++ '',++ 'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',++ 'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',++ 'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',++ 'GET /api/health, /api/ready',++ '',++ 'Source: https://github.com/OpenVibers/OpenVibe.Sources',++ '',++ ].join('\n'));++ });++++ app.use((req, res) => http.sendProblem(res, 404, 'sources.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));++++ // eslint-disable-next-line no-unused-vars++ app.use((err, req, res, _next) => {++ if (err.type === 'entity.parse.failed') return http.sendProblem(res, 400, 'sources.bad_json', { detail: 'request body is not valid JSON', ctx: req.ov });++ if (err.type === 'entity.too.large') return http.sendProblem(res, 413, 'sources.too_large', { detail: 'request body too large', ctx: req.ov });++ log.error(`[app] ${req.method} ${req.path}: ${err.stack || err}`);++ if (res.headersSent) return res.end();++ return http.sendProblem(res, 500, 'sources.internal', { detail: 'internal error', ctx: req.ov });++ });++++ return app;++}++++module.exports = { createApp };nnew file mode 100644@@@ -0,0 +1,104 @@++'use strict';++/**++ * Service-token authentication and capability checks. Every API route is for services (other++ * OpenVibe services, or the Network admin surface acting through its own principal): an++ * RS256 client-credentials token from OpenVibe.Network (audience openvibe.sources), verified++ * offline, and ONE capability per route.++ *++ * The sources.* capabilities are proposed in docs/capabilities-proposal/ and are not in++ * openvibe-contracts v0.7.0 yet; checkCapability() decides them with the library's own grant rule++ * (exact id or a `family.*` grant) until a release knows them, then the library decides.++ */++const crypto = require('crypto');++const { serviceAuth, capabilities, http } = require('openvibe-contracts');++++const CAPS = Object.freeze({++ read: 'sources.source.read',++ items: 'sources.item.read',++ manage: 'sources.source.manage',++});++const PROPOSED = new Set(Object.values(CAPS));++++function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, log = console } = {}) {++ let key = pem ? toPem(pem) : null;++ let retryTimer = null;++ let refreshTimer = null;++++ function toPem(value) {++ return crypto.createPublicKey(value).export({ type: 'spki', format: 'pem' });++ }++++ async function fetchOnce() {++ for (const base of urls) {++ if (!base) continue;++ const url = `${base}/api/.well-known/jwks`;++ try {++ const res = await fetchImpl(url, { signal: AbortSignal.timeout(8000) });++ if (!res.ok) throw new Error(`HTTP ${res.status}`);++ const body = await res.json();++ const jwk = (body.keys || []).find(k => k.kty === 'RSA');++ if (jwk) key = toPem({ key: jwk, format: 'jwk' });++ else if (typeof body.public_key === 'string' && body.public_key.includes('BEGIN')) key = toPem(body.public_key);++ else throw new Error('no RSA key in response');++ log.log(`[auth] Network public key loaded from ${base}`);++ return key;++ } catch (err) {++ log.warn(`[auth] key fetch from ${url} failed: ${err.message}`);++ }++ }++ return null;++ }++++ function start() {++ if (pem) return Promise.resolve(key);++ const attempt = async () => {++ const k = await fetchOnce();++ if (!k && !key) {++ retryTimer = setTimeout(attempt, 30 * 1000);++ retryTimer.unref?.();++ }++ return k;++ };++ refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);++ refreshTimer.unref?.();++ return attempt();++ }++++ function stop() {++ clearTimeout(retryTimer);++ clearInterval(refreshTimer);++ }++++ return { get: () => key, loaded: () => Boolean(key), start, stop };++}++++function checkCapability(claims, capabilityId) {++ if (!capabilities.get(capabilityId) && PROPOSED.has(capabilityId)) {++ return capabilities.grants(claims && claims.cap, capabilityId)++ ? { allowed: true, code: null, reason: null }++ : { allowed: false, code: 'capability.denied', reason: `${capabilityId} not granted` };++ }++ return capabilities.check(claims, capabilityId);++}++++function createAuth({ config, keys }) {++ /** Express guard: one capability. Sets req.principal = { sub, cap, jti }. */++ function requireCap(id) {++ return function capGuard(req, res, next) {++ const ctx = req.ov;++ const h = String(req.headers.authorization || '');++ if (!h.startsWith('Bearer ')) return http.sendProblem(res, 401, 'token.missing', { detail: 'a service token is required', ctx });++ const publicKey = keys.get();++ if (!publicKey) return http.sendProblem(res, 503, 'token.unavailable', { detail: 'signing key not loaded yet', ctx });++ const r = serviceAuth.verifyServiceToken(h.slice(7).trim(), { publicKey, issuer: config.issuer, audience: config.audience });++ if (!r.ok) return http.sendProblem(res, 401, r.code, { detail: r.reason, ctx });++ const c = checkCapability(r.claims, id);++ if (!c.allowed) return http.sendProblem(res, 403, c.code, { detail: c.reason, ctx });++ req.principal = { sub: r.claims.sub, cap: r.claims.cap, jti: r.claims.jti };++ return next();++ };++ }++ return { requireCap };++}++++module.exports = { CAPS, PROPOSED, createKeyStore, createAuth, checkCapability };nnew file mode 100644@@@ -0,0 +1,80 @@++'use strict';++/**++ * OpenVibe.Sources configuration. Everything comes from the environment (.env in development,++ * /etc/openvibe/sources.env in production); see .env.example for the documented list.++ *++ * load(env) is pure so tests can build a config without touching process.env. `env` is also kept++ * as config.secrets: a source's credential is read from it by environment-variable NAME at fetch++ * time and never stored, logged or returned.++ */++const pkg = require('../package.json');++++const int = (v, d) => {++ const n = parseInt(v, 10);++ return Number.isFinite(n) ? n : d;++};++const list = (v, d) => (v == null || v === '' ? d : String(v).split(',').map(s => s.trim()).filter(Boolean));++const strip = (v) => String(v || '').replace(/\/$/, '');++++function load(env = process.env) {++ const nodeEnv = env.NODE_ENV || 'development';++ const isProduction = nodeEnv === 'production';++ const port = int(env.PORT, 4720);++ return {++ port,++ host: env.HOST || '127.0.0.1',++ nodeEnv,++ isProduction,++ serviceId: 'sources',++ baseUrl: strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`)),++++ networkUrl: strip(env.OV_NETWORK_URL || 'https://openvibe.network'),++ networkInternalUrl: strip(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),++ issuer: strip(env.OV_NETWORK_ISSUER || env.OV_NETWORK_URL || 'https://openvibe.network'),++ networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,++ audience: 'openvibe.sources',++ oauth: {++ clientId: env.OV_OAUTH_CLIENT_ID || 'sources',++ clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',++ },++++ dbPath: env.SOURCES_DB_PATH || './data/sources.db',++++ events: {++ url: strip(env.EVENTS_URL || ''),++ relayIntervalMs: int(env.EVENTS_RELAY_INTERVAL_MS, 2000),++ },++++ // Ingestion++ worker: {++ enabled: env.SOURCES_WORKER !== 'off',++ tickMs: int(env.SOURCES_TICK_MS, 5000),++ maxConcurrent: int(env.SOURCES_MAX_CONCURRENT, 4),++ // Longest wait between polls of a failing source (exponential backoff cap).++ maxBackoffMs: int(env.SOURCES_MAX_BACKOFF_MS, 6 * 3600 * 1000),++ },++ fetch: {++ userAgent: env.SOURCES_USER_AGENT || `OpenVibeSources/${pkg.version} (+https://sources.openvibe.network; [email protected])`,++ // Product token matched against robots.txt User-agent lines.++ robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeSources',++ timeoutMs: int(env.SOURCES_FETCH_TIMEOUT_MS, 20000),++ maxBytes: int(env.SOURCES_MAX_BYTES, 5 * 1024 * 1024),++ maxRedirects: int(env.SOURCES_MAX_REDIRECTS, 5),++ robotsMaxBytes: 512 * 1024,++ robotsTtlMs: int(env.SOURCES_ROBOTS_TTL_MS, 24 * 3600 * 1000),++ // Minimum gap between any two requests to one host, whatever the sources say.++ hostMinIntervalMs: int(env.SOURCES_HOST_MIN_INTERVAL_MS, 1000),++ allowedPorts: list(env.SOURCES_ALLOWED_PORTS, ['80', '443']).map(Number),++ // Exact hostnames/IPs that may resolve to private or loopback addresses (tests and++ // deliberate on-host sources only). Empty in production.++ allowPrivateHosts: list(env.SOURCES_ALLOW_PRIVATE_HOSTS, []),++ },++ items: {++ maxPerFetch: int(env.SOURCES_MAX_ITEMS_PER_FETCH, 500),++ summaryMax: 1000,++ },++ secrets: env,++ };++}++++module.exports = { load };nnew file mode 100644@@@ -0,0 +1,158 @@++'use strict';++/**++ * The Sources database (SQLite, WAL). One file owned by this service only; created on boot,++ * idempotently.++ */++const fs = require('fs');++const path = require('path');++const Database = require('better-sqlite3');++const outbox = require('./events/outbox');++++const SCHEMA = `++CREATE TABLE IF NOT EXISTS sources (++ key TEXT PRIMARY KEY,++ name TEXT NOT NULL,++ type TEXT NOT NULL, -- rss | atom | sitemap | jsonld | api | manual++ category TEXT NOT NULL, -- news | blog | reviews | deals | coupons | trade++ homepage_url TEXT,++ endpoints TEXT NOT NULL DEFAULT '[]',++ auth TEXT NOT NULL DEFAULT '{"mode":"none"}', -- env var NAME only, never a value++ robots_note TEXT,++ terms_note TEXT,++ license_note TEXT,++ min_interval_ms INTEGER NOT NULL, -- rate limit: gap between two requests to this source++ poll_interval_sec INTEGER NOT NULL,++ stale_after_sec INTEGER NOT NULL,++ max_items INTEGER NOT NULL,++ enabled INTEGER NOT NULL DEFAULT 0,++ review_required INTEGER NOT NULL DEFAULT 1,++ sensitivity TEXT NOT NULL DEFAULT 'none',++ default_indexability TEXT NOT NULL DEFAULT 'noindex',++ search_visibility TEXT, -- null = items are not sent to OpenVibe.Search++ created_at INTEGER NOT NULL,++ updated_at INTEGER NOT NULL,++ updated_by TEXT,++ -- runtime state++ next_due_at INTEGER NOT NULL DEFAULT 0,++ not_before INTEGER NOT NULL DEFAULT 0, -- Retry-After / Crawl-delay++ last_request_at INTEGER,++ last_run_at INTEGER,++ last_success_at INTEGER,++ last_state TEXT,++ consecutive_failures INTEGER NOT NULL DEFAULT 0++);++++CREATE TABLE IF NOT EXISTS endpoint_state (++ source_key TEXT NOT NULL,++ url TEXT NOT NULL,++ etag TEXT,++ last_modified TEXT,++ last_status INTEGER,++ last_fetch_at INTEGER,++ PRIMARY KEY (source_key, url)++);++++-- One row per fetch attempt, whatever happened. state is never empty.++CREATE TABLE IF NOT EXISTS fetch_runs (++ rid INTEGER PRIMARY KEY AUTOINCREMENT,++ id TEXT NOT NULL UNIQUE,++ source_key TEXT NOT NULL,++ endpoint_url TEXT,++ trigger TEXT NOT NULL, -- schedule | manual++ started_at INTEGER NOT NULL,++ finished_at INTEGER NOT NULL,++ state TEXT NOT NULL CHECK (state IN ('ok','not_modified','http_error','timeout','robots_denied','parse_error','rate_limited','disabled')),++ http_status INTEGER,++ error_code TEXT,++ detail TEXT,++ bytes INTEGER,++ raw_body_hash TEXT,++ parser_version TEXT,++ items_seen INTEGER NOT NULL DEFAULT 0,++ items_created INTEGER NOT NULL DEFAULT 0,++ items_updated INTEGER NOT NULL DEFAULT 0,++ items_unchanged INTEGER NOT NULL DEFAULT 0,++ items_skipped INTEGER NOT NULL DEFAULT 0++);++CREATE INDEX IF NOT EXISTS idx_fetch_runs_source ON fetch_runs (source_key, rid);++++CREATE TABLE IF NOT EXISTS items (++ rid INTEGER PRIMARY KEY AUTOINCREMENT,++ id TEXT NOT NULL UNIQUE,++ source_key TEXT NOT NULL,++ category TEXT NOT NULL,++ identity TEXT NOT NULL, -- guid / @id / url as the source gave it++ kind TEXT NOT NULL,++ canonical_url TEXT,++ title TEXT,++ summary TEXT,++ authors TEXT NOT NULL DEFAULT '[]',++ published_at TEXT,++ source_updated_at TEXT,++ fields TEXT NOT NULL DEFAULT '{}',++ content_hash TEXT NOT NULL,++ raw_body_hash TEXT,++ parser_version TEXT NOT NULL,++ license_note TEXT,++ terms_note TEXT,++ first_seen_at INTEGER NOT NULL,++ retrieved_at INTEGER NOT NULL, -- last successful fetch that contained it++ revision INTEGER NOT NULL DEFAULT 1,++ last_fetch_run_id TEXT,++ entered_by TEXT, -- manual items: the principal that entered it++ removed_at INTEGER,++ removed_reason TEXT,++ change_seq INTEGER NOT NULL,++ UNIQUE (source_key, identity)++);++CREATE INDEX IF NOT EXISTS idx_items_change ON items (change_seq);++CREATE INDEX IF NOT EXISTS idx_items_source ON items (source_key, change_seq);++CREATE INDEX IF NOT EXISTS idx_items_category ON items (category, change_seq);++++CREATE TABLE IF NOT EXISTS item_revisions (++ item_id TEXT NOT NULL,++ revision INTEGER NOT NULL,++ content_hash TEXT NOT NULL,++ raw_body_hash TEXT,++ parser_version TEXT NOT NULL,++ fetch_run_id TEXT,++ retrieved_at INTEGER NOT NULL,++ snapshot TEXT NOT NULL,++ PRIMARY KEY (item_id, revision)++);++++CREATE TABLE IF NOT EXISTS counters (++ name TEXT PRIMARY KEY,++ value INTEGER NOT NULL++);++++CREATE TABLE IF NOT EXISTS robots_cache (++ origin TEXT PRIMARY KEY,++ fetched_at INTEGER NOT NULL,++ expires_at INTEGER NOT NULL,++ status INTEGER,++ outcome TEXT NOT NULL, -- parsed | unavailable (4xx: allow) | unreachable (deny)++ rules TEXT NOT NULL DEFAULT '[]',++ crawl_delay_sec REAL,++ detail TEXT++);++`;++++function openDb(dbPath) {++ if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });++ const db = new Database(dbPath);++ db.pragma('journal_mode = WAL');++ db.pragma('synchronous = NORMAL');++ db.pragma('busy_timeout = 5000');++ db.exec(SCHEMA);++ outbox.ensureSchema(db);++ return db;++}++++/** Next value of a named monotonic counter (call inside a transaction). */++function nextSeq(db, name) {++ db.prepare('INSERT INTO counters (name, value) VALUES (?, 1) ON CONFLICT(name) DO UPDATE SET value = value + 1').run(name);++ return db.prepare('SELECT value FROM counters WHERE name = ?').get(name).value;++}++++module.exports = { openDb, nextSeq };nnew file mode 100644@@@ -0,0 +1,179 @@++'use strict';++/**++ * Transactional outbox (events.event-envelope@1), OpenVibe.Events client semantics:++ *++ * enqueue(envelope) INSIDE the transaction that makes the change, so an event exists exactly++ * when its effect does (throws outside a transaction)++ * relay publishes unsent rows to OpenVibe.Events (POST /api/v1/events, service++ * token for audience openvibe.events with events.event.publish), marks them++ * sent, backs off on failure; a permanently refused row (4xx other than++ * 401/408/429) is marked rejected so it cannot block the rest++ *++ * The relay runs only when EVENTS_URL is set; without it rows wait and are relayed later. Events++ * deduplicates on event_id, so a retry after a lost response never publishes twice.++ */++const { ids, validate, serviceAuth } = require('openvibe-contracts');++++const BACKOFF_MS = [1000, 5000, 30000, 120000, 600000];++++function ensureSchema(db) {++ db.exec(`CREATE TABLE IF NOT EXISTS event_outbox (++ seq INTEGER PRIMARY KEY AUTOINCREMENT,++ event_id TEXT NOT NULL UNIQUE,++ event_type TEXT NOT NULL,++ envelope TEXT NOT NULL,++ created_at INTEGER NOT NULL,++ attempts INTEGER NOT NULL DEFAULT 0,++ next_attempt_at INTEGER NOT NULL DEFAULT 0,++ sent_at INTEGER,++ rejected_at INTEGER,++ last_error TEXT++ );++ CREATE INDEX IF NOT EXISTS idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);`);++}++++function createOutbox(db, { source, now = () => Date.now() }) {++ const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');++++ /** { event_type, subject, payload, visibility?, priority?, actor?, trace_id? } → envelope */++ function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {++ if (!db.inTransaction) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');++ const ms = now();++ const env = {++ event_id: ids.newId('event', ms),++ event_type,++ version: 1,++ source,++ actor: actor || { type: 'service', id: source },++ timestamp: new Date(ms).toISOString(),++ priority,++ visibility,++ subject,++ payload: payload || {},++ };++ if (trace_id && /^[0-9a-f]{32}$/.test(trace_id)) env.trace_id = trace_id;++ const v = validate('events.event-envelope@1', env);++ if (!v.valid) throw new Error(`outbox: invalid envelope for ${event_type}: ${v.errors.map(e => `${e.path} ${e.message}`).join('; ')}`);++ insert.run(env.event_id, event_type, JSON.stringify(env), ms);++ return env;++ }++++ const q = {++ due: db.prepare('SELECT seq, event_id, envelope, attempts FROM event_outbox WHERE sent_at IS NULL AND rejected_at IS NULL AND next_attempt_at <= ? ORDER BY seq LIMIT ?'),++ sent: db.prepare('UPDATE event_outbox SET sent_at = ?, attempts = attempts + 1, last_error = NULL WHERE seq = ?'),++ failed: db.prepare('UPDATE event_outbox SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE seq = ?'),++ rejected: db.prepare('UPDATE event_outbox SET rejected_at = ?, attempts = attempts + 1, last_error = ? WHERE seq = ?'),++ pending: db.prepare('SELECT COUNT(*) AS n FROM event_outbox WHERE sent_at IS NULL AND rejected_at IS NULL'),++ rejectedCount: db.prepare('SELECT COUNT(*) AS n FROM event_outbox WHERE rejected_at IS NOT NULL'),++ list: db.prepare('SELECT envelope FROM event_outbox ORDER BY seq'),++ prune: db.prepare('DELETE FROM event_outbox WHERE sent_at IS NOT NULL AND sent_at < ?'),++ };++++ return {++ enqueue,++ pending: () => q.pending.get().n,++ rejected: () => q.rejectedCount.get().n,++ /** Every envelope in order (tests and operators). */++ all: () => q.list.all().map(r => JSON.parse(r.envelope)),++ prune: (olderThanMs = 7 * 24 * 3600 * 1000) => q.prune.run(now() - olderThanMs).changes,++ _q: q,++ };++}++++/**++ * createRelay({ db, outbox, eventsUrl, intervalMs?, tokenClient? | tokenOpts?, fetchImpl?, log?, now? })++ * → { start, stop, flush, running }++ * tokenOpts = { tokenUrl, clientId, clientSecret } builds a client-credentials token client for++ * audience openvibe.events. Without eventsUrl or a token source the relay does nothing.++ */++function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenOpts, fetchImpl = globalThis.fetch, log = console, now = () => Date.now(), db }) {++ const q = outbox._q;++ const tokens = tokenClient || (tokenOpts && serviceAuth.createTokenClient({ ...tokenOpts, audience: 'openvibe.events', scope: 'events.event.publish', fetchImpl }));++ let timer = null;++ let flushing = null;++++ function permanent(status) {++ return status >= 400 && status < 500 && ![401, 408, 429].includes(status);++ }++++ async function post(rows) {++ const body = rows.length === 1 ? JSON.parse(rows[0].envelope) : { events: rows.map(r => JSON.parse(r.envelope)) };++ const res = await fetchImpl(`${eventsUrl}/api/v1/events`, {++ method: 'POST',++ headers: { 'Content-Type': 'application/json', Accept: 'application/json', ...(await tokens.authHeaders()) },++ body: JSON.stringify(body),++ signal: AbortSignal.timeout(10000),++ });++ if (res.status === 401 && typeof tokens.invalidate === 'function') tokens.invalidate();++ const text = await res.text().catch(() => '');++ return { ok: res.ok, status: res.status, text };++ }++++ function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }++++ async function publishRows(rows) {++ let r;++ try {++ r = await post(rows);++ } catch (err) {++ mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));++ return { sent: 0, failed: rows.length, rejected: 0 };++ }++ if (r.ok) {++ mark(rows, row => q.sent.run(now(), row.seq));++ return { sent: rows.length, failed: 0, rejected: 0 };++ }++ if (permanent(r.status) && rows.length > 1) {++ // One bad envelope refuses the batch: isolate it.++ const total = { sent: 0, failed: 0, rejected: 0 };++ for (const row of rows) {++ const s = await publishRows([row]);++ total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;++ }++ return total;++ }++ const msg = `${r.status} ${r.text.slice(0, 300)}`;++ if (permanent(r.status)) {++ mark(rows, row => q.rejected.run(now(), msg, row.seq));++ log.warn(`[outbox] event refused by Events: ${msg}`);++ return { sent: 0, failed: 0, rejected: rows.length };++ }++ mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));++ return { sent: 0, failed: rows.length, rejected: 0 };++ }++++ async function doFlush() {++ const total = { sent: 0, failed: 0, rejected: 0 };++ if (!eventsUrl || !tokens) return total;++ for (;;) {++ const rows = q.due.all(now(), 50);++ if (!rows.length) break;++ const s = await publishRows(rows);++ total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;++ if (s.failed || rows.length < 50) break;++ }++ if (total.failed) log.warn(`[outbox] ${total.failed} event(s) not published yet; will retry`);++ return total;++ }++++ function flush() {++ if (!flushing) flushing = doFlush().finally(() => { flushing = null; });++ return flushing;++ }++++ function start() {++ if (timer || !eventsUrl) return;++ timer = setInterval(() => { flush().catch(err => log.warn(`[outbox] relay: ${err.message}`)); }, intervalMs);++ timer.unref?.();++ }++++ function stop() {++ if (timer) clearInterval(timer);++ timer = null;++ return flushing || Promise.resolve();++ }++++ return { start, stop, flush, running: () => Boolean(timer) };++}++++module.exports = { ensureSchema, createOutbox, createRelay };nnew file mode 100644@@@ -0,0 +1,94 @@++'use strict';++/**++ * OpenVibe.Sources entry point.++ *++ * node server/index.js (systemd: openvibe-sources.service)++ *++ * start() is also what the tests use: it takes a config (server/config.js load()) plus injectable++ * clock/fetch/log/DNS lookup, and returns handles to every part.++ */++const { load } = require('./config');++const { openDb } = require('./db');++const { createRegistry } = require('./registry');++const { createItems } = require('./items');++const { createGuard } = require('./net/guard');++const { createFetcher } = require('./net/fetcher');++const { createSpacer } = require('./spacer');++const { createRobots } = require('./robots');++const { createIngest } = require('./ingest');++const { createScheduler } = require('./scheduler');++const { createOutbox, createRelay } = require('./events/outbox');++const { createKeyStore, createAuth } = require('./auth');++const { createApp } = require('./app');++++async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, lookupImpl, log = console, listen = true } = {}) {++ config = config || load();++ const db = openDb(config.dbPath);++ const outbox = createOutbox(db, { source: config.serviceId, now });++ const relay = createRelay({++ db, outbox, eventsUrl: config.events.url, intervalMs: config.events.relayIntervalMs, fetchImpl, log, now,++ tokenClient,++ tokenOpts: config.oauth.clientSecret ? { tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret } : null,++ });++ const registry = createRegistry({ db, now, maxItemsCap: config.items.maxPerFetch });++ const items = createItems({ db, outbox, now });++ const guard = createGuard({ allowPrivateHosts: config.fetch.allowPrivateHosts, allowedPorts: config.fetch.allowedPorts, ...(lookupImpl ? { lookupImpl } : {}) });++ const fetcher = createFetcher({ guard, userAgent: config.fetch.userAgent, timeoutMs: config.fetch.timeoutMs, maxBytes: config.fetch.maxBytes, maxRedirects: config.fetch.maxRedirects });++ const spacer = createSpacer({ hostMinIntervalMs: config.fetch.hostMinIntervalMs });++ const robots = createRobots({++ db, fetcher, agent: config.fetch.robotsAgent, ttlMs: config.fetch.robotsTtlMs, maxBytes: config.fetch.robotsMaxBytes, now,++ beforeRequest: (host) => spacer.space(host, 0),++ });++ const ingest = createIngest({ db, config, registry, items, robots, fetcher, spacer, outbox, now, log, relay });++ const scheduler = createScheduler({ db, ingest, config, now, log });++ const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });++ const auth = createAuth({ config, keys });++ const app = createApp({ config, db, registry, items, ingest, scheduler, auth, keys, outbox, relay, now, log });++++ const keyLoaded = keys.start().catch(() => null);++ relay.start();++ if (config.worker.enabled) scheduler.start();++ const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);++ pruneTimer.unref?.();++++ let server = null;++ if (listen) {++ server = await new Promise((resolve, reject) => {++ const s = app.listen(config.port, config.host, () => resolve(s));++ s.on('error', reject);++ });++ log.log(`[sources] listening on http://${config.host}:${server.address().port}`);++ }++++ async function close() {++ clearInterval(pruneTimer);++ keys.stop();++ await scheduler.stop();++ await relay.stop();++ if (server) {++ server.closeAllConnections?.();++ await new Promise(resolve => server.close(() => resolve()));++ }++ db.close();++ }++++ return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, relay, keys, keyLoaded, auth, app, server, close };++}++++if (require.main === module) {++ require('dotenv').config();++ start().then((handles) => {++ const shutdown = (sig) => {++ console.log(`[sources] ${sig}: shutting down`);++ handles.close().then(() => process.exit(0), () => process.exit(1));++ setTimeout(() => process.exit(1), 20000).unref();++ };++ process.on('SIGTERM', () => shutdown('SIGTERM'));++ process.on('SIGINT', () => shutdown('SIGINT'));++ }).catch((err) => {++ console.error(`[sources] failed to start: ${err.stack || err}`);++ process.exit(1);++ });++}++++module.exports = { start };nnew file mode 100644@@@ -0,0 +1,268 @@++'use strict';++/**++ * Ingestion runs: one source at a time (never two runs of the same source in flight), each++ * endpoint fetched in order, every attempt recorded as a fetch_runs row with an explicit state:++ *++ * ok fetched and parsed; items created/updated/unchanged++ * not_modified 304 to a conditional GET (ETag / Last-Modified from the last good parse)++ * http_error any other status, or a refused/failed connection, oversize body, bad redirect++ * timeout no complete response within SOURCES_FETCH_TIMEOUT_MS++ * robots_denied robots.txt disallows the URL (or a redirect target), or is unreachable++ * parse_error the body is not what the adapter reads++ * rate_limited the source's rate limit or a Retry-After has not elapsed (nothing requested),++ * or the site answered 429++ * disabled the source is disabled, or its credential variable is not set++ *++ * A failed run never creates or modifies an item: only the `ok` path calls items.ingest(), in the++ * same transaction as its run row. Validators (ETag/Last-Modified) are stored only after a good++ * parse, so a body that failed to parse is fetched and parsed again next time, not skipped as 304.++ */++const { ids } = require('openvibe-contracts');++const { ADAPTERS } = require('./adapters');++const { sha256, decodeBody } = require('./adapters/util');++++const FAILURE_STATES = new Set(['http_error', 'timeout', 'robots_denied', 'parse_error', 'rate_limited']);++++function retryAfterMs(value, now) {++ if (value == null) return null;++ const s = String(value).trim();++ if (/^\d{1,7}$/.test(s)) return Number(s) * 1000;++ const t = Date.parse(s);++ return Number.isFinite(t) ? Math.max(0, t - now) : null;++}++++function createIngest({ db, config, registry, items, robots, fetcher, spacer, outbox, now = () => Date.now(), log = console, relay = null }) {++ const inflight = new Set();++ const { space, setCrawlDelay } = spacer;++++ const st = {++ insertRun: db.prepare(`INSERT INTO fetch_runs (id, source_key, endpoint_url, trigger, started_at, finished_at, state, http_status,++ error_code, detail, bytes, raw_body_hash, parser_version, items_seen, items_created, items_updated, items_unchanged, items_skipped)++ VALUES (@id, @source_key, @endpoint_url, @trigger, @started_at, @finished_at, @state, @http_status, @error_code, @detail, @bytes,++ @raw_body_hash, @parser_version, @items_seen, @items_created, @items_updated, @items_unchanged, @items_skipped)`),++ endpoint: db.prepare('SELECT * FROM endpoint_state WHERE source_key = ? AND url = ?'),++ saveValidators: db.prepare(`INSERT INTO endpoint_state (source_key, url, etag, last_modified, last_status, last_fetch_at) VALUES (@k, @u, @etag, @lm, @status, @at)++ ON CONFLICT(source_key, url) DO UPDATE SET etag = excluded.etag, last_modified = excluded.last_modified, last_status = excluded.last_status, last_fetch_at = excluded.last_fetch_at`),++ touchEndpoint: db.prepare(`INSERT INTO endpoint_state (source_key, url, last_status, last_fetch_at) VALUES (@k, @u, @status, @at)++ ON CONFLICT(source_key, url) DO UPDATE SET last_status = excluded.last_status, last_fetch_at = excluded.last_fetch_at`),++ requested: db.prepare('UPDATE sources SET last_request_at = ? WHERE key = ?'),++ finish: db.prepare(`UPDATE sources SET last_run_at = @at, last_state = @state, consecutive_failures = @failures,++ last_success_at = COALESCE(@success_at, last_success_at), next_due_at = @next_due, not_before = @not_before WHERE key = @key`),++ };++++ // ── Run rows ─────────────────────────────────────────────++++ function baseRun(source, endpointUrl, trigger, startedAt) {++ return {++ id: `frn_${ids.ulid(startedAt)}`, source_key: source.key, endpoint_url: endpointUrl, trigger,++ started_at: startedAt, finished_at: now(), state: null, http_status: null, error_code: null, detail: null,++ bytes: null, raw_body_hash: null, parser_version: null,++ items_seen: 0, items_created: 0, items_updated: 0, items_unchanged: 0, items_skipped: 0,++ };++ }++++ function failedEvent(source, run, failures) {++ outbox.enqueue({++ event_type: 'sources.fetch.failed',++ subject: { type: 'source', id: source.key },++ payload: {++ source_key: source.key, category: source.category, run_id: run.id, endpoint_url: run.endpoint_url,++ state: run.state, error_code: run.error_code, http_status: run.http_status, consecutive_failures: failures,++ },++ });++ }++++ /** Record a run that changed no item (every non-ok outcome, and 304). */++ function recordNoItems(source, run, { countsAsFailure, failures }) {++ db.transaction(() => {++ st.insertRun.run(run);++ if (run.endpoint_url && run.http_status != null) st.touchEndpoint.run({ k: source.key, u: run.endpoint_url, status: run.http_status, at: run.finished_at });++ if (countsAsFailure) failedEvent(source, run, failures);++ })();++ return run;++ }++++ // ── One endpoint ─────────────────────────────────────────++++ async function fetchEndpoint(source, endpoint, trigger) {++ const adapter = ADAPTERS[source.type];++ const startedAt = now();++ const run = baseRun(source, endpoint.url, trigger, startedAt);++ const target = new URL(endpoint.url);++ const host = target.host;++++ // credential by variable name, read now, never stored++ const headers = { Accept: adapter.accept };++ const sensitive = [];++ let requestUrl = target;++ if (source.auth.mode !== 'none') {++ const secret = config.secrets[source.auth.env];++ if (!secret) {++ Object.assign(run, { state: 'disabled', error_code: 'credential_missing', detail: `environment variable ${source.auth.env} is not set`, finished_at: now() });++ return { run, failed: false };++ }++ if (source.auth.mode === 'bearer') { headers.Authorization = `Bearer ${secret}`; sensitive.push('Authorization'); }++ if (source.auth.mode === 'header') { headers[source.auth.header] = secret; sensitive.push(source.auth.header); }++ if (source.auth.mode === 'query') { requestUrl = new URL(target); requestUrl.searchParams.set(source.auth.param, secret); }++ }++++ // robots.txt for the endpoint's origin (a robots fetch is a request to the host too)++ const verdict = await robots.check(target.toString());++ setCrawlDelay(host, verdict.crawlDelaySec);++ if (!verdict.allowed) {++ if (verdict.refusedCode) Object.assign(run, { state: 'http_error', error_code: verdict.refusedCode, detail: `${host} is not fetchable (${verdict.refusedCode})`, finished_at: now() });++ else Object.assign(run, { state: 'robots_denied', error_code: 'robots', detail: verdict.reason, finished_at: now() });++ return { run, failed: true };++ }++++ const validators = st.endpoint.get(source.key, endpoint.url) || {};++ await space(host, source.min_interval_ms);++ st.requested.run(now(), source.key);++ let res;++ try {++ res = await fetcher.fetchUrl(requestUrl.toString(), {++ headers,++ etag: validators.etag || null,++ lastModified: validators.last_modified || null,++ sensitiveHeaders: sensitive,++ // every redirect hop is checked against its own origin's robots.txt and, on a new++ // host, spaced like any other request to that host++ beforeHop: async (next) => {++ const v = await robots.check(next.toString());++ if (!v.allowed) return { refuse: v.reason };++ if (next.host !== host) {++ setCrawlDelay(next.host, v.crawlDelaySec);++ await space(next.host, source.min_interval_ms);++ }++ return null;++ },++ });++ } catch (err) {++ const state = err.code === 'timeout' ? 'timeout' : err.code === 'hop_refused' ? 'robots_denied' : 'http_error';++ Object.assign(run, { state, error_code: err.code || 'network', detail: String(err.message).slice(0, 500), finished_at: now() });++ return { run, failed: true };++ }++ run.http_status = res.status;++ run.bytes = res.bytes != null ? res.bytes : null;++ run.finished_at = now();++++ if (res.status === 304) {++ Object.assign(run, { state: 'not_modified' });++ return { run, failed: false, success: true };++ }++ if (res.status === 429) {++ const wait = Math.min(retryAfterMs(res.headers['retry-after'], now()) ?? 10 * 60 * 1000, 24 * 3600 * 1000);++ Object.assign(run, { state: 'rate_limited', error_code: 'upstream_429', detail: `the site asked us to wait ${Math.round(wait / 1000)} s` });++ return { run, failed: true, notBefore: now() + wait };++ }++ if (res.status !== 200 && res.status !== 203) {++ const wait = res.status === 503 ? retryAfterMs(res.headers['retry-after'], now()) : null;++ Object.assign(run, { state: 'http_error', error_code: `http_${res.status}`, detail: `HTTP ${res.status}` });++ return { run, failed: true, notBefore: wait != null ? now() + Math.min(wait, 24 * 3600 * 1000) : null };++ }++++ run.raw_body_hash = sha256(res.body);++ run.parser_version = adapter.version;++ let parsed;++ try {++ const text = decodeBody(res.body, res.headers['content-type']);++ parsed = adapter.parse(text, { url: res.finalUrl, endpoint, summaryMax: config.items.summaryMax, maxItems: source.max_items });++ } catch (err) {++ Object.assign(run, { state: 'parse_error', error_code: err.code === 'parse_error' ? 'unreadable' : 'adapter_error', detail: String(err.message).slice(0, 500) });++ if (err.code !== 'parse_error') log.error(`[ingest] ${source.key}: adapter crashed: ${err.stack || err}`);++ return { run, failed: true };++ }++++ // The only path that touches items: the run row, items, validators and events commit together.++ db.transaction(() => {++ const counts = items.ingest(source, parsed.items, {++ runId: run.id, rawBodyHash: run.raw_body_hash, parserVersion: adapter.version, retrievedAt: run.finished_at,++ });++ Object.assign(run, {++ state: 'ok', items_seen: parsed.items.length, items_created: counts.created, items_updated: counts.updated,++ items_unchanged: counts.unchanged, items_skipped: parsed.skipped + counts.skipped,++ });++ st.insertRun.run(run);++ st.saveValidators.run({++ k: source.key, u: endpoint.url, etag: res.headers.etag || null, lm: res.headers['last-modified'] || null,++ status: res.status, at: run.finished_at,++ });++ })();++ return { run, failed: false, success: true, recorded: true };++ }++++ // ── One source ───────────────────────────────────────────++++ /**++ * run(key, { trigger }) → { runs: [...] } | { busy: true } | { manual: true } | null (unknown)++ */++ async function run(key, { trigger = 'schedule' } = {}) {++ const row = registry.get(key);++ if (!row) return null;++ if (row.type === 'manual') return { manual: true };++ if (inflight.has(key)) return { busy: true };++ inflight.add(key);++ try {++ const source = registry.fromRow(row);++ const t = now();++ const runs = [];++++ if (!source.enabled) {++ const r = baseRun(source, null, trigger, t);++ Object.assign(r, { state: 'disabled', error_code: 'source_disabled', detail: 'the source is disabled' });++ runs.push(recordNoItems(source, r, { countsAsFailure: false }));++ return { runs };++ }++ const sinceLast = row.last_request_at == null ? Infinity : t - row.last_request_at;++ if (sinceLast < source.min_interval_ms || t < row.not_before) {++ const r = baseRun(source, null, trigger, t);++ const until = Math.max(row.not_before, (row.last_request_at || 0) + source.min_interval_ms);++ Object.assign(r, { state: 'rate_limited', error_code: 'local_rate_limit', detail: `next request allowed at ${new Date(until).toISOString()}` });++ runs.push(recordNoItems(source, r, { countsAsFailure: false }));++ return { runs, rateLimitedUntil: until };++ }++++ let anyFailure = false;++ let anySuccess = false;++ let notBefore = 0;++ let failures = row.consecutive_failures;++ for (const endpoint of source.endpoints) {++ let out;++ try {++ out = await fetchEndpoint(source, endpoint, trigger);++ } catch (err) {++ // Anything unexpected is still a recorded failure, never a silent gap.++ const r = baseRun(source, endpoint.url, trigger, t);++ Object.assign(r, { state: 'http_error', error_code: 'internal', detail: String(err.message).slice(0, 300), finished_at: now() });++ log.error(`[ingest] ${key}: ${err.stack || err}`);++ out = { run: r, failed: true };++ }++ if (out.failed) { anyFailure = true; failures = row.consecutive_failures + 1; }++ if (out.success) anySuccess = true;++ if (out.notBefore) notBefore = Math.max(notBefore, out.notBefore);++ if (!out.recorded) recordNoItems(source, out.run, { countsAsFailure: out.failed, failures });++ runs.push(out.run);++ if (out.run.state === 'rate_limited') break; // the site asked us to stop++ }++++ const finishedAt = now();++ const newFailures = anyFailure ? row.consecutive_failures + 1 : 0;++ const backoff = anyFailure++ ? Math.min(source.poll_interval_sec * 1000 * 2 ** Math.min(newFailures - 1, 10), Math.max(config.worker.maxBackoffMs, source.poll_interval_sec * 1000))++ : source.poll_interval_sec * 1000;++ const lastState = runs.map(r => r.state).find(s => FAILURE_STATES.has(s)) || runs[runs.length - 1].state;++ st.finish.run({++ key, at: finishedAt, state: lastState, failures: newFailures,++ success_at: anySuccess ? finishedAt : null, next_due: finishedAt + backoff, not_before: notBefore,++ });++ if (relay) relay.flush().catch(() => {});++ return { runs };++ } finally {++ inflight.delete(key);++ }++ }++++ return { run, inflight: () => [...inflight] };++}++++module.exports = { createIngest, FAILURE_STATES, retryAfterMs };nnew file mode 100644@@@ -0,0 +1,239 @@++'use strict';++/**++ * Source items: provenance-carrying records of what a source said, when we retrieved it, and++ * from which exact body and parser.++ *++ * Every item stores its source key, canonical URL, retrieval time (first seen, last seen), a++ * content hash of the parsed fields, the hash of the raw body it was parsed from, the parser++ * version and the source's license/terms notes as they were at ingestion. A changed content hash++ * is a new revision (the previous one stays in item_revisions); an unchanged item only has its++ * retrieval time refreshed. Removal (a takedown, a licence problem) is explicit, audited and++ * sticky: a later fetch that still lists the item does not bring it back.++ *++ * Only successful fetches reach ingest(); nothing in this module runs for a failed fetch.++ */++const { ids } = require('openvibe-contracts');++const { nextSeq } = require('./db');++const { sha256, canonicalJson } = require('./adapters/util');++++const STAFF_GROUPS = ['role:admin', 'role:global_mod'];++++function contentHash(it) {++ return sha256(canonicalJson({++ kind: it.kind, canonical_url: it.canonical_url || null, title: it.title || null, summary: it.summary || null,++ authors: it.authors || [], published_at: it.published_at || null, updated_at: it.updated_at || null, fields: it.fields || {},++ }));++}++++function createItems({ db, outbox, now = () => Date.now() }) {++ const st = {++ byIdentity: db.prepare('SELECT * FROM items WHERE source_key = ? AND identity = ?'),++ byId: db.prepare('SELECT * FROM items WHERE id = ?'),++ insert: db.prepare(`INSERT INTO items (id, source_key, category, identity, kind, canonical_url, title, summary, authors, published_at,++ source_updated_at, fields, content_hash, raw_body_hash, parser_version, license_note, terms_note, first_seen_at, retrieved_at,++ revision, last_fetch_run_id, entered_by, change_seq)++ VALUES (@id, @source_key, @category, @identity, @kind, @canonical_url, @title, @summary, @authors, @published_at,++ @source_updated_at, @fields, @content_hash, @raw_body_hash, @parser_version, @license_note, @terms_note, @first_seen_at, @retrieved_at,++ 1, @last_fetch_run_id, @entered_by, @change_seq)`),++ update: db.prepare(`UPDATE items SET kind = @kind, canonical_url = @canonical_url, title = @title, summary = @summary, authors = @authors,++ published_at = @published_at, source_updated_at = @source_updated_at, fields = @fields, content_hash = @content_hash,++ raw_body_hash = @raw_body_hash, parser_version = @parser_version, license_note = @license_note, terms_note = @terms_note,++ retrieved_at = @retrieved_at, revision = revision + 1, last_fetch_run_id = @last_fetch_run_id, change_seq = @change_seq++ WHERE id = @id`),++ seen: db.prepare('UPDATE items SET retrieved_at = ?, last_fetch_run_id = ? WHERE id = ?'),++ remove: db.prepare('UPDATE items SET removed_at = ?, removed_reason = ?, revision = revision + 1, change_seq = ? WHERE id = ?'),++ revision: db.prepare(`INSERT INTO item_revisions (item_id, revision, content_hash, raw_body_hash, parser_version, fetch_run_id, retrieved_at, snapshot)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?)`),++ revisions: db.prepare('SELECT * FROM item_revisions WHERE item_id = ? ORDER BY revision'),++ };++++ function view(row) {++ if (!row) return null;++ const iso = (v) => (v == null ? null : new Date(v).toISOString());++ return {++ id: row.id,++ source_key: row.source_key,++ category: row.category,++ kind: row.kind,++ identity: row.identity,++ canonical_url: row.canonical_url,++ title: row.title,++ summary: row.summary,++ authors: JSON.parse(row.authors),++ published_at: row.published_at,++ source_updated_at: row.source_updated_at,++ fields: JSON.parse(row.fields),++ revision: row.revision,++ provenance: {++ retrieved_at: iso(row.retrieved_at),++ first_seen_at: iso(row.first_seen_at),++ content_hash: row.content_hash,++ raw_body_hash: row.raw_body_hash,++ parser_version: row.parser_version,++ fetch_run_id: row.last_fetch_run_id,++ license_note: row.license_note,++ terms_note: row.terms_note,++ entered_by: row.entered_by,++ },++ removed: row.removed_at ? { at: iso(row.removed_at), reason: row.removed_reason } : null,++ change_seq: row.change_seq,++ };++ }++++ function subjectOf(row) {++ return { type: 'item', id: row.id, revision: row.revision };++ }++++ function itemEvent(type, row, extra = {}) {++ outbox.enqueue({++ event_type: type,++ subject: subjectOf(row),++ payload: {++ item_id: row.id, source_key: row.source_key, category: row.category, kind: row.kind,++ canonical_url: row.canonical_url, title: row.title, revision: row.revision, content_hash: row.content_hash,++ retrieved_at: new Date(row.retrieved_at).toISOString(), ...extra,++ },++ });++ }++++ /** search.index-document@1 for a raw item: staff-only, never indexable, provenance attached. */++ function indexDocument(row) {++ const title = row.title || row.canonical_url || row.identity;++ const doc = {++ owner: 'sources', type: 'item', id: row.id, revision: row.revision,++ visibility: 'members', acl: { groups: STAFF_GROUPS },++ title: String(title).slice(0, 500),++ summary: row.summary ? row.summary.slice(0, 4000) : '',++ body: '',++ facets: { category: row.category, source: row.source_key, kind: row.kind },++ authorship: 'imported',++ provenance: [{++ service: 'sources', type: 'item', id: row.id, revision: row.revision,++ ...(row.canonical_url ? { url: row.canonical_url } : {}),++ retrieved_at: new Date(row.retrieved_at).toISOString(),++ }],++ publication_state: 'published',++ published_at: row.published_at || null,++ updated_at: row.source_updated_at || null,++ indexability: { decision: 'noindex', reasons: ['third_party_content'] },++ };++ if (row.canonical_url) doc.canonical_url = row.canonical_url;++ return doc;++ }++++ function indexEvent(source, row, deleted = false) {++ if (!source.search_visibility) return;++ outbox.enqueue({++ event_type: deleted ? 'sources.index_document.deleted' : 'sources.index_document.upserted',++ subject: subjectOf(row),++ payload: deleted ? { type: 'item', id: row.id, revision: row.revision } : indexDocument(row),++ });++ }++++ function snapshot(row) {++ return JSON.stringify({++ kind: row.kind, canonical_url: row.canonical_url, title: row.title, summary: row.summary,++ authors: JSON.parse(row.authors), published_at: row.published_at, source_updated_at: row.source_updated_at, fields: JSON.parse(row.fields),++ });++ }++++ function values(source, it, ctx) {++ return {++ kind: it.kind,++ canonical_url: it.canonical_url || null,++ title: it.title || null,++ summary: it.summary || null,++ authors: JSON.stringify(it.authors || []),++ published_at: it.published_at || null,++ source_updated_at: it.updated_at || null,++ fields: JSON.stringify(it.fields || {}),++ content_hash: contentHash(it),++ raw_body_hash: ctx.rawBodyHash || null,++ parser_version: ctx.parserVersion,++ license_note: source.license_note || null,++ terms_note: source.terms_note || null,++ retrieved_at: ctx.retrievedAt,++ last_fetch_run_id: ctx.runId || null,++ };++ }++++ /**++ * Apply one successful fetch's parsed items. Must run inside the transaction that also writes++ * the fetch run row. → { created, updated, unchanged, skipped }++ */++ function ingest(source, parsedItems, ctx) {++ if (!db.inTransaction) throw new Error('items.ingest() must run inside the fetch run transaction');++ const counts = { created: 0, updated: 0, unchanged: 0, skipped: 0 };++ for (const it of parsedItems) {++ const cur = st.byIdentity.get(source.key, it.identity);++ const v = values(source, it, ctx);++ if (!cur) {++ const row = { id: `itm_${ids.ulid(ctx.retrievedAt)}`, source_key: source.key, category: source.category, identity: it.identity, first_seen_at: ctx.retrievedAt, entered_by: ctx.enteredBy || null, change_seq: nextSeq(db, 'items'), ...v };++ st.insert.run(row);++ const saved = st.byId.get(row.id);++ st.revision.run(saved.id, 1, saved.content_hash, saved.raw_body_hash, saved.parser_version, ctx.runId || null, ctx.retrievedAt, snapshot(saved));++ itemEvent('sources.item.created', saved);++ indexEvent(source, saved);++ counts.created++;++ } else if (cur.removed_at) {++ counts.skipped++; // removal is sticky++ } else if (cur.content_hash !== v.content_hash) {++ st.update.run({ ...v, id: cur.id, change_seq: nextSeq(db, 'items') });++ const saved = st.byId.get(cur.id);++ st.revision.run(saved.id, saved.revision, saved.content_hash, saved.raw_body_hash, saved.parser_version, ctx.runId || null, ctx.retrievedAt, snapshot(saved));++ itemEvent('sources.item.updated', saved, { previous_content_hash: cur.content_hash });++ indexEvent(source, saved);++ counts.updated++;++ } else {++ st.seen.run(ctx.retrievedAt, ctx.runId || null, cur.id);++ counts.unchanged++;++ }++ }++ return counts;++ }++++ /** Remove an item (takedown, licence, error). Sticky; emits sources.item.removed. */++ function remove(source, id, reason, by) {++ return db.transaction(() => {++ const cur = st.byId.get(id);++ if (!cur || cur.source_key !== source.key) return null;++ if (cur.removed_at) return view(cur);++ const t = now();++ st.remove.run(t, `${reason}${by ? ` (by ${by})` : ''}`.slice(0, 500), nextSeq(db, 'items'), id);++ const saved = st.byId.get(id);++ st.revision.run(saved.id, saved.revision, saved.content_hash, saved.raw_body_hash, saved.parser_version, null, t, JSON.stringify({ removed: true, reason }));++ itemEvent('sources.item.removed', saved, { reason });++ indexEvent(source, saved, true);++ return view(saved);++ })();++ }++++ function get(id, { revisions = false } = {}) {++ const row = st.byId.get(id);++ if (!row) return null;++ const out = view(row);++ if (revisions) {++ out.revisions = st.revisions.all(id).map(r => ({++ revision: r.revision, content_hash: r.content_hash, raw_body_hash: r.raw_body_hash, parser_version: r.parser_version,++ fetch_run_id: r.fetch_run_id, retrieved_at: new Date(r.retrieved_at).toISOString(), snapshot: JSON.parse(r.snapshot),++ }));++ }++ return out;++ }++++ /** Items in change order (created, updated or removed after `after`). */++ function list({ source = null, category = null, after = 0, limit = 100, includeRemoved = false } = {}) {++ const rows = db.prepare(`SELECT * FROM items WHERE change_seq > @after++ AND (@source IS NULL OR source_key = @source) AND (@category IS NULL OR category = @category)++ AND (@incl = 1 OR removed_at IS NULL)++ ORDER BY change_seq LIMIT @limit`).all({ after, source, category, incl: includeRemoved ? 1 : 0, limit: limit + 1 });++ const more = rows.length > limit;++ const page = rows.slice(0, limit);++ // next_after is where to resume, also when this page is the last one (poll with it later).++ return { items: page.map(view), next_after: page.length ? page[page.length - 1].change_seq : after, more };++ }++++ return { ingest, remove, get, list, view, indexDocument, contentHash };++}++++module.exports = { createItems, contentHash, STAFF_GROUPS };nnew file mode 100644@@@ -0,0 +1,116 @@++'use strict';++/**++ * The only way Sources reaches the internet: GET with an address guard (server/net/guard.js),++ * manual redirects re-checked hop by hop, conditional headers, one overall deadline covering++ * connect + headers + body, a byte cap applied to the decoded body, and no cookies.++ *++ * fetchUrl(url, { headers, etag, lastModified, beforeHop(url) }) →++ * { kind: 'response', status, headers, body (Buffer|null), finalUrl, bytes }++ * or throws FetchError { code: timeout | too_large | address_refused | port_refused | bad_url |++ * too_many_redirects | network | hop_refused, detail }++ */++const http = require('http');++const https = require('https');++const zlib = require('zlib');++const { GuardError } = require('./guard');++++class FetchError extends Error {++ constructor(code, detail) { super(detail); this.code = code; }++}++++function createFetcher({ guard, userAgent, timeoutMs = 20000, maxBytes = 5 * 1024 * 1024, maxRedirects = 5 }) {++ function once(url, headers, deadline, capBytes) {++ return new Promise((resolve, reject) => {++ const mod = url.protocol === 'https:' ? https : http;++ const remaining = deadline - Date.now();++ if (remaining <= 0) return reject(new FetchError('timeout', 'deadline passed'));++ let settled = false;++ const done = (fn, v) => { if (!settled) { settled = true; clearTimeout(timer); fn(v); } };++ const req = mod.request(url, {++ method: 'GET',++ headers,++ lookup: guard.lookup,++ agent: false,++ timeout: remaining,++ }, (res) => {++ const status = res.statusCode;++ if (status >= 300 && status < 400 && res.headers.location) {++ res.resume();++ return done(resolve, { redirect: res.headers.location, status, headers: res.headers });++ }++ const declared = Number(res.headers['content-length']);++ if (Number.isFinite(declared) && declared > capBytes && status === 200) {++ req.destroy();++ return done(reject, new FetchError('too_large', `content-length ${declared} exceeds ${capBytes} bytes`));++ }++ let stream = res;++ const enc = String(res.headers['content-encoding'] || '').trim().toLowerCase();++ if (enc === 'gzip' || enc === 'x-gzip') stream = res.pipe(zlib.createGunzip());++ else if (enc === 'deflate') stream = res.pipe(zlib.createInflate());++ else if (enc === 'br') stream = res.pipe(zlib.createBrotliDecompress());++ const chunks = [];++ let bytes = 0;++ stream.on('data', (c) => {++ bytes += c.length;++ if (bytes > capBytes) {++ req.destroy();++ done(reject, new FetchError('too_large', `body exceeds ${capBytes} bytes`));++ return;++ }++ chunks.push(c);++ });++ stream.on('end', () => done(resolve, { status, headers: res.headers, body: Buffer.concat(chunks), bytes }));++ stream.on('error', (err) => done(reject, new FetchError('network', `body: ${err.message}`)));++ res.on('aborted', () => done(reject, new FetchError('network', 'response aborted')));++ });++ const timer = setTimeout(() => {++ req.destroy();++ done(reject, new FetchError('timeout', `no complete response within ${timeoutMs} ms`));++ }, remaining);++ req.on('timeout', () => {++ req.destroy();++ done(reject, new FetchError('timeout', `no complete response within ${timeoutMs} ms`));++ });++ req.on('error', (err) => {++ if (err instanceof GuardError) return done(reject, new FetchError(err.code, err.message));++ done(reject, new FetchError('network', `${err.code || 'error'}: ${err.message}`));++ });++ req.end();++ });++ }++++ async function fetchUrl(input, { headers = {}, etag = null, lastModified = null, beforeHop = null, capBytes = maxBytes, sensitiveHeaders = [] } = {}) {++ const deadline = Date.now() + timeoutMs;++ let url;++ try { url = guard.checkUrl(input); } catch (err) { throw new FetchError(err.code || 'bad_url', err.message); }++ const h = {++ 'User-Agent': userAgent,++ Accept: headers.Accept || '*/*',++ 'Accept-Encoding': 'gzip, deflate, br',++ ...headers,++ };++ if (etag) h['If-None-Match'] = etag;++ if (lastModified) h['If-Modified-Since'] = lastModified;++ for (let hop = 0; ; hop++) {++ const r = await once(url, h, deadline, capBytes);++ if (!r.redirect) return { kind: 'response', status: r.status, headers: r.headers, body: r.body, bytes: r.bytes, finalUrl: url.toString() };++ if (hop >= maxRedirects) throw new FetchError('too_many_redirects', `more than ${maxRedirects} redirects`);++ let next;++ try { next = guard.checkUrl(new URL(r.redirect, url).toString()); } catch (err) { throw new FetchError(err.code || 'bad_url', `redirect: ${err.message}`); }++ if (beforeHop) {++ const verdict = await beforeHop(next);++ if (verdict && verdict.refuse) throw new FetchError('hop_refused', verdict.refuse);++ }++ // Conditional headers belong to the original resource only; credentials never leave++ // the origin they were configured for.++ delete h['If-None-Match'];++ delete h['If-Modified-Since'];++ if (next.origin !== url.origin) for (const name of sensitiveHeaders) delete h[name];++ url = next;++ }++ }++++ return { fetchUrl };++}++++module.exports = { createFetcher, FetchError };nnew file mode 100644@@@ -0,0 +1,114 @@++'use strict';++/**++ * Outbound address policy (roadmap anti-goal 13: no fetcher may reach internal addresses).++ *++ * A URL is fetchable only when it is http(s), on an allowed port, and every address its host++ * resolves to is public unicast. The check runs at connect time through the socket's `lookup`++ * (so a DNS answer cannot change between check and connect) and up front for IP literals (which++ * never go through lookup). SOURCES_ALLOW_PRIVATE_HOSTS lists exact hostnames/IPs exempted —++ * for tests and deliberate on-host sources only.++ */++const dns = require('dns');++const net = require('net');++++const blocked = new net.BlockList();++for (const [addr, prefix] of [++ ['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8], ['169.254.0.0', 16],++ ['172.16.0.0', 12], ['192.0.0.0', 24], ['192.0.2.0', 24], ['192.88.99.0', 24], ['192.168.0.0', 16],++ ['198.18.0.0', 15], ['198.51.100.0', 24], ['203.0.113.0', 24], ['224.0.0.0', 4], ['240.0.0.0', 4],++]) blocked.addSubnet(addr, prefix, 'ipv4');++for (const [addr, prefix] of [++ ['::', 128], ['::1', 128], ['100::', 64], ['2001:db8::', 32], ['2001::', 32], ['fc00::', 7],++ ['fe80::', 10], ['fec0::', 10], ['ff00::', 8],++]) blocked.addSubnet(addr, prefix, 'ipv6');++++class GuardError extends Error {++ constructor(code, detail) { super(detail); this.code = code; }++}++++/** IPv4 embedded in IPv4-mapped (::ffff:a.b.c.d), NAT64 (64:ff9b::/96) or 6to4 (2002::/16). */++function embeddedV4(ip) {++ const lower = ip.toLowerCase();++ const mapped = /^::ffff:(\d+\.\d+\.\d+\.\d+)$/.exec(lower);++ if (mapped) return mapped[1];++ const full = expandV6(lower);++ if (!full) return null;++ const hex = (i) => parseInt(full[i], 16);++ const v4 = (hi, lo) => `${hi >> 8}.${hi & 255}.${lo >> 8}.${lo & 255}`;++ if (full.slice(0, 5).every(h => h === '0000') && full[5] === 'ffff') return v4(hex(6), hex(7));++ if (full[0] === '0064' && full[1] === 'ff9b' && full.slice(2, 6).every(h => h === '0000')) return v4(hex(6), hex(7));++ if (full[0] === '2002') return v4(hex(1), hex(2));++ return null;++}++++function expandV6(ip) {++ if (!net.isIPv6(ip)) return null;++ let s = ip;++ const v4tail = /(\d+\.\d+\.\d+\.\d+)$/.exec(s);++ if (v4tail) {++ const p = v4tail[1].split('.').map(Number);++ s = s.slice(0, -v4tail[1].length) + ((p[0] << 8) | p[1]).toString(16) + ':' + ((p[2] << 8) | p[3]).toString(16);++ }++ const [head, tail] = s.split('::');++ const h = head ? head.split(':') : [];++ const t = tail !== undefined ? (tail ? tail.split(':') : []) : [];++ const fill = tail !== undefined ? Array(8 - h.length - t.length).fill('0') : [];++ const parts = [...h, ...fill, ...t];++ if (parts.length !== 8) return null;++ return parts.map(x => x.padStart(4, '0'));++}++++/** Is this IP address public unicast? */++function isPublicAddress(ip) {++ const family = net.isIP(ip);++ if (family === 4) return !blocked.check(ip, 'ipv4');++ if (family === 6) {++ const v4 = embeddedV4(ip);++ if (v4) return !blocked.check(v4, 'ipv4');++ return !blocked.check(ip, 'ipv6');++ }++ return false;++}++++function createGuard({ allowPrivateHosts = [], allowedPorts = [80, 443], lookupImpl = dns.lookup } = {}) {++ const exempt = new Set(allowPrivateHosts.map(h => h.toLowerCase().replace(/^\[|\]$/g, '')));++++ /** Synchronous URL checks: scheme, credentials, port, IP literal. Throws GuardError. */++ function checkUrl(input) {++ let url;++ try { url = new URL(input); } catch { throw new GuardError('bad_url', 'not a URL'); }++ if (url.protocol !== 'http:' && url.protocol !== 'https:') throw new GuardError('bad_url', `scheme ${url.protocol} is not fetched`);++ if (url.username || url.password) throw new GuardError('bad_url', 'credentials in URLs are not allowed');++ const host = url.hostname.toLowerCase().replace(/^\[|\]$/g, '');++ const port = url.port ? Number(url.port) : (url.protocol === 'https:' ? 443 : 80);++ if (!exempt.has(host) && !allowedPorts.includes(port)) throw new GuardError('port_refused', `port ${port} is not allowed`);++ if (net.isIP(host) && !exempt.has(host) && !isPublicAddress(host)) throw new GuardError('address_refused', `${host} is not a public address`);++ return url;++ }++++ /** dns.lookup replacement for http(s).request: refuses non-public answers at connect time. */++ function lookup(hostname, options, callback) {++ if (typeof options === 'function') { callback = options; options = {}; }++ const opts = typeof options === 'number' ? { family: options } : { ...(options || {}) };++ const host = String(hostname).toLowerCase();++ lookupImpl(hostname, { ...opts, all: true }, (err, addresses) => {++ if (err) return callback(err);++ const list = Array.isArray(addresses) ? addresses : [{ address: addresses, family: net.isIP(addresses) }];++ if (!list.length) return callback(Object.assign(new Error(`no address for ${hostname}`), { code: 'ENOTFOUND' }));++ if (!exempt.has(host)) {++ const bad = list.find(a => !isPublicAddress(a.address));++ if (bad) {++ const e = new GuardError('address_refused', `${hostname} resolves to a non-public address`);++ e.errno = 'EADDRREFUSED';++ return callback(e);++ }++ }++ if (opts.all) return callback(null, list);++ return callback(null, list[0].address, list[0].family);++ });++ }++++ return { checkUrl, lookup, isPublicAddress };++}++++module.exports = { createGuard, isPublicAddress, GuardError, embeddedV4 };nnew file mode 100644@@@ -0,0 +1,240 @@++'use strict';++/**++ * The source registry: validation, CRUD, and the health/staleness view.++ *++ * A source records where it is fetched from and on what terms: endpoints, the auth mode with the++ * NAME of the environment variable holding the credential (never its value; names must start with++ * SOURCES_CRED_ so a registry entry can never point at this service's own secrets), robots and++ * terms notes, the rate limit, whether it is enabled, review/sensitivity flags and the default++ * indexability downstream products start from. A source cannot be enabled without a terms note:++ * an adapter existing is not permission to ingest (roadmap anti-goal 26).++ */++const { checkApiMapping } = require('./adapters');++++const TYPES = ['rss', 'atom', 'sitemap', 'jsonld', 'api', 'manual'];++const CATEGORIES = ['news', 'blog', 'reviews', 'deals', 'coupons', 'trade'];++const SENSITIVITY = ['none', 'financial', 'health', 'political', 'legal', 'adult'];++const AUTH_MODES = ['none', 'header', 'bearer', 'query'];++const KEY_RE = /^[a-z0-9][a-z0-9-]{1,63}$/;++const ENV_RE = /^SOURCES_CRED_[A-Z0-9_]{2,60}$/;++const HEADER_RE = /^[A-Za-z][A-Za-z0-9-]{0,63}$/;++const FORBIDDEN_HEADERS = new Set(['host', 'cookie', 'connection', 'content-length', 'transfer-encoding', 'user-agent', 'accept-encoding', 'if-none-match', 'if-modified-since']);++++class RegistryError extends Error {++ constructor(detail, code = 'sources.bad_source') { super(detail); this.code = code; }++}++++function httpUrl(v, what) {++ let u;++ try { u = new URL(String(v)); } catch { throw new RegistryError(`${what} is not a URL`); }++ if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new RegistryError(`${what} must be http(s)`);++ if (u.username || u.password) throw new RegistryError(`${what} must not carry credentials`);++ if (String(v).length > 2048) throw new RegistryError(`${what} is too long`);++ return u.toString();++}++++const text = (v, max, what) => {++ if (v == null || v === '') return null;++ if (typeof v !== 'string' || v.length > max) throw new RegistryError(`${what} must be a string of at most ${max} characters`);++ return v.trim() || null;++};++const intIn = (v, min, max, d, what) => {++ if (v === undefined || v === null) return d;++ if (!Number.isInteger(v) || v < min || v > max) throw new RegistryError(`${what} must be an integer between ${min} and ${max}`);++ return v;++};++const bool = (v, d, what) => {++ if (v === undefined || v === null) return d;++ if (typeof v !== 'boolean') throw new RegistryError(`${what} must be true or false`);++ return v;++};++++/** Validate a full source definition → normalized record (no runtime state). */++function validateSource(input, { maxItemsCap = 500 } = {}) {++ if (!input || typeof input !== 'object' || Array.isArray(input)) throw new RegistryError('body must be a source object');++ const known = new Set(['key', 'name', 'type', 'category', 'homepage_url', 'endpoints', 'auth', 'robots_note', 'terms_note', 'license_note',++ 'min_interval_ms', 'poll_interval_sec', 'stale_after_sec', 'max_items', 'enabled', 'review_required', 'sensitivity',++ 'default_indexability', 'search_visibility']);++ for (const k of Object.keys(input)) if (!known.has(k)) throw new RegistryError(`unknown field ${k}`);++ if (!KEY_RE.test(String(input.key || ''))) throw new RegistryError('key must match ^[a-z0-9][a-z0-9-]{1,63}$');++ const name = text(input.name, 200, 'name');++ if (!name) throw new RegistryError('name is required');++ if (!TYPES.includes(input.type)) throw new RegistryError(`type must be one of ${TYPES.join('|')}`);++ if (!CATEGORIES.includes(input.category)) throw new RegistryError(`category must be one of ${CATEGORIES.join('|')}`);++++ const endpoints = input.endpoints == null ? [] : input.endpoints;++ if (!Array.isArray(endpoints) || endpoints.length > 20) throw new RegistryError('endpoints must be an array of at most 20');++ if (input.type === 'manual' && endpoints.length) throw new RegistryError('a manual source has no endpoints');++ if (input.type !== 'manual' && !endpoints.length) throw new RegistryError('at least one endpoint is required');++ const eps = endpoints.map((ep, i) => {++ const e = typeof ep === 'string' ? { url: ep } : ep;++ if (!e || typeof e !== 'object') throw new RegistryError(`endpoints[${i}] must be a URL or an object`);++ const out = { url: httpUrl(e.url, `endpoints[${i}].url`) };++ if (input.type === 'api') {++ const err = checkApiMapping(e);++ if (err) throw new RegistryError(`endpoints[${i}]: ${err}`);++ Object.assign(out, { format: e.format || 'json', items_path: e.items_path || '', item_kind: e.item_kind || 'record', fields: e.fields || {}, extra: e.extra || {} });++ } else {++ for (const k of Object.keys(e)) if (k !== 'url') throw new RegistryError(`endpoints[${i}].${k} is only for api sources`);++ }++ return out;++ });++ if (new Set(eps.map(e => e.url)).size !== eps.length) throw new RegistryError('endpoints must be distinct');++++ const a = input.auth == null ? { mode: 'none' } : input.auth;++ if (typeof a !== 'object' || !AUTH_MODES.includes(a.mode)) throw new RegistryError(`auth.mode must be one of ${AUTH_MODES.join('|')}`);++ const auth = { mode: a.mode };++ if (a.mode !== 'none') {++ if (input.type === 'manual') throw new RegistryError('a manual source has no credential');++ if (!ENV_RE.test(String(a.env || ''))) throw new RegistryError('auth.env must be an environment variable NAME matching ^SOURCES_CRED_[A-Z0-9_]+$ (never a value)');++ auth.env = a.env;++ if (a.mode === 'header') {++ if (!HEADER_RE.test(String(a.header || '')) || FORBIDDEN_HEADERS.has(String(a.header).toLowerCase())) throw new RegistryError('auth.header must be a plain header name');++ auth.header = a.header;++ }++ if (a.mode === 'query') {++ if (!/^[A-Za-z0-9_.-]{1,64}$/.test(String(a.param || ''))) throw new RegistryError('auth.param must be a query parameter name');++ auth.param = a.param;++ }++ }++ for (const k of Object.keys(a)) if (!['mode', 'env', 'header', 'param'].includes(k)) throw new RegistryError(`unknown auth field ${k}`);++++ const pollIntervalSec = intIn(input.poll_interval_sec, 1, 7 * 86400, 3600, 'poll_interval_sec');++ const record = {++ key: input.key,++ name,++ type: input.type,++ category: input.category,++ homepage_url: input.homepage_url ? httpUrl(input.homepage_url, 'homepage_url') : null,++ endpoints: eps,++ auth,++ robots_note: text(input.robots_note, 2000, 'robots_note'),++ terms_note: text(input.terms_note, 2000, 'terms_note'),++ license_note: text(input.license_note, 2000, 'license_note'),++ min_interval_ms: intIn(input.min_interval_ms, 0, 86400000, 60000, 'min_interval_ms'),++ poll_interval_sec: pollIntervalSec,++ stale_after_sec: intIn(input.stale_after_sec, 1, 90 * 86400, pollIntervalSec * 3, 'stale_after_sec'),++ max_items: intIn(input.max_items, 1, maxItemsCap, Math.min(200, maxItemsCap), 'max_items'),++ enabled: bool(input.enabled, false, 'enabled'),++ review_required: bool(input.review_required, true, 'review_required'),++ sensitivity: input.sensitivity == null ? 'none' : input.sensitivity,++ default_indexability: input.default_indexability == null ? 'noindex' : input.default_indexability,++ search_visibility: input.search_visibility == null ? null : input.search_visibility,++ };++ if (!SENSITIVITY.includes(record.sensitivity)) throw new RegistryError(`sensitivity must be one of ${SENSITIVITY.join('|')}`);++ if (!['index', 'noindex'].includes(record.default_indexability)) throw new RegistryError('default_indexability must be index or noindex');++ if (record.sensitivity !== 'none' && record.default_indexability === 'index' && record.review_required === false) {++ throw new RegistryError('a sensitive source needs review before anything from it can be indexable');++ }++ if (![null, 'members'].includes(record.search_visibility)) throw new RegistryError('search_visibility must be null or "members" (staff-only search of raw items)');++ if (record.enabled && !record.terms_note) throw new RegistryError('a source cannot be enabled without a terms_note (access and terms must be verified first)', 'sources.terms_required');++ if (record.enabled && record.type !== 'manual' && !record.robots_note) throw new RegistryError('a fetched source cannot be enabled without a robots_note', 'sources.terms_required');++ return record;++}++++function createRegistry({ db, now = () => Date.now(), maxItemsCap = 500 }) {++ const st = {++ get: db.prepare('SELECT * FROM sources WHERE key = ?'),++ all: db.prepare('SELECT * FROM sources ORDER BY key'),++ insert: db.prepare(`INSERT INTO sources (key, name, type, category, homepage_url, endpoints, auth, robots_note, terms_note, license_note,++ min_interval_ms, poll_interval_sec, stale_after_sec, max_items, enabled, review_required, sensitivity, default_indexability,++ search_visibility, created_at, updated_at, updated_by, next_due_at)++ VALUES (@key, @name, @type, @category, @homepage_url, @endpoints, @auth, @robots_note, @terms_note, @license_note,++ @min_interval_ms, @poll_interval_sec, @stale_after_sec, @max_items, @enabled, @review_required, @sensitivity, @default_indexability,++ @search_visibility, @created_at, @updated_at, @updated_by, 0)`),++ update: db.prepare(`UPDATE sources SET name = @name, type = @type, category = @category, homepage_url = @homepage_url, endpoints = @endpoints,++ auth = @auth, robots_note = @robots_note, terms_note = @terms_note, license_note = @license_note, min_interval_ms = @min_interval_ms,++ poll_interval_sec = @poll_interval_sec, stale_after_sec = @stale_after_sec, max_items = @max_items, enabled = @enabled,++ review_required = @review_required, sensitivity = @sensitivity, default_indexability = @default_indexability,++ search_visibility = @search_visibility, updated_at = @updated_at, updated_by = @updated_by WHERE key = @key`),++ del: db.prepare('DELETE FROM sources WHERE key = ?'),++ itemCount: db.prepare('SELECT COUNT(*) AS n FROM items WHERE source_key = ?'),++ };++++ function toRow(rec, by) {++ return {++ ...rec,++ endpoints: JSON.stringify(rec.endpoints),++ auth: JSON.stringify(rec.auth),++ enabled: rec.enabled ? 1 : 0,++ review_required: rec.review_required ? 1 : 0,++ created_at: now(),++ updated_at: now(),++ updated_by: by || null,++ };++ }++++ function fromRow(row) {++ if (!row) return null;++ return {++ key: row.key, name: row.name, type: row.type, category: row.category, homepage_url: row.homepage_url,++ endpoints: JSON.parse(row.endpoints), auth: JSON.parse(row.auth),++ robots_note: row.robots_note, terms_note: row.terms_note, license_note: row.license_note,++ min_interval_ms: row.min_interval_ms, poll_interval_sec: row.poll_interval_sec, stale_after_sec: row.stale_after_sec,++ max_items: row.max_items, enabled: Boolean(row.enabled), review_required: Boolean(row.review_required),++ sensitivity: row.sensitivity, default_indexability: row.default_indexability, search_visibility: row.search_visibility,++ };++ }++++ /** Health and staleness, computed from the runtime columns. */++ function health(row) {++ const t = now();++ const neverSucceeded = row.last_success_at == null;++ const stale = row.type === 'manual' ? false : neverSucceeded || (t - row.last_success_at) > row.stale_after_sec * 1000;++ let status;++ if (!row.enabled) status = 'disabled';++ else if (row.type === 'manual') status = 'manual';++ else if (row.last_run_at == null) status = 'never_fetched';++ else if (row.consecutive_failures > 0) status = 'failing';++ else if (stale) status = 'stale';++ else status = 'healthy';++ const iso = (v) => (v == null ? null : new Date(v).toISOString());++ return {++ status,++ stale,++ last_success_at: iso(row.last_success_at),++ last_run_at: iso(row.last_run_at),++ last_state: row.last_state,++ consecutive_failures: row.consecutive_failures,++ next_due_at: row.enabled && row.type !== 'manual' ? iso(Math.max(row.next_due_at, row.not_before)) : null,++ stale_after_sec: row.stale_after_sec,++ };++ }++++ function view(row) {++ return { ...fromRow(row), health: health(row), created_at: new Date(row.created_at).toISOString(), updated_at: new Date(row.updated_at).toISOString() };++ }++++ function create(input, by) {++ const rec = validateSource(input, { maxItemsCap });++ if (st.get.get(rec.key)) throw new RegistryError(`source ${rec.key} exists`, 'sources.exists');++ st.insert.run(toRow(rec, by));++ return st.get.get(rec.key);++ }++++ function patch(key, changes, by) {++ const row = st.get.get(key);++ if (!row) return null;++ if (changes && changes.key !== undefined && changes.key !== key) throw new RegistryError('key cannot change');++ if (changes && changes.type !== undefined && changes.type !== row.type && st.itemCount.get(key).n) {++ throw new RegistryError('type cannot change once the source has items (register a new source)');++ }++ const merged = { ...fromRow(row), ...(changes || {}), key };++ const rec = validateSource(merged, { maxItemsCap });++ st.update.run({ ...toRow(rec, by) });++ return st.get.get(key);++ }++++ /** Only a source that never produced an item can be deleted; otherwise disable it. */++ function remove(key) {++ const row = st.get.get(key);++ if (!row) return null;++ if (st.itemCount.get(key).n) throw new RegistryError('the source has items; disable it instead (provenance must stay resolvable)', 'sources.has_items');++ st.del.run(key);++ return true;++ }++++ return { get: (k) => st.get.get(k), all: () => st.all.all(), create, patch, remove, view, fromRow, health };++}++++module.exports = { createRegistry, validateSource, RegistryError, TYPES, CATEGORIES };nnew file mode 100644@@@ -0,0 +1,147 @@++'use strict';++/**++ * robots.txt (RFC 9309), checked before every request an ingestion run makes, including each++ * redirect hop to a new origin.++ *++ * - the group whose User-agent names our product token (case-insensitive) applies, else `*`;++ * several matching groups are merged++ * - Allow/Disallow with `*` and `$`; the longest matching rule wins, Allow wins a tie++ * - 2xx: parsed; 4xx (e.g. 404) except 429: no restrictions; 429, 5xx, network error or timeout:++ * complete disallow ("unreachable") — never guessed as allowed++ * - cached per origin for SOURCES_ROBOTS_TTL_MS (24 h); Crawl-delay is honoured as a minimum++ * gap between requests (non-standard, but a clear wish of the site)++ * A source-adapter name is never permission to fetch (roadmap anti-goal 26): if robots says no,++ * the fetch run is `robots_denied` and nothing is requested.++ */++++function parseRobots(text, agent) {++ const token = String(agent).toLowerCase();++ const groups = [];++ let current = null;++ let lastWasAgent = false;++ for (const rawLine of String(text).split(/\r\n|\r|\n/)) {++ const line = rawLine.replace(/#.*$/, '').trim();++ if (!line) continue;++ const i = line.indexOf(':');++ if (i < 0) continue;++ const key = line.slice(0, i).trim().toLowerCase();++ const value = line.slice(i + 1).trim();++ if (key === 'user-agent') {++ if (!lastWasAgent || !current) {++ current = { agents: [], rules: [], crawlDelay: null };++ groups.push(current);++ }++ current.agents.push(value.toLowerCase());++ lastWasAgent = true;++ continue;++ }++ lastWasAgent = false;++ if (!current) continue;++ // Bounded: a hostile robots.txt cannot make matching expensive.++ if ((key === 'allow' || key === 'disallow') && value.length <= 512 && current.rules.length < 1000) {++ current.rules.push({ allow: key === 'allow', path: value.replace(/\*+/g, '*') });++ }++ else if (key === 'crawl-delay') {++ const n = Number(value);++ if (Number.isFinite(n) && n >= 0) current.crawlDelay = Math.min(n, 3600);++ }++ }++ // RFC 9309: the product token is matched case-insensitively against the User-agent value.++ let chosen = groups.filter(g => g.agents.some(a => a.split('/')[0].trim() === token));++ if (!chosen.length) chosen = groups.filter(g => g.agents.includes('*'));++ const delays = chosen.map(g => g.crawlDelay).filter(d => d != null);++ return {++ // An empty Disallow (or Allow) path matches nothing: "allow everything".++ rules: chosen.flatMap(g => g.rules).filter(r => r.path !== ''),++ crawlDelaySec: delays.length ? Math.max(...delays) : null,++ };++}++++function ruleRegex(path) {++ let p = path;++ const anchored = p.endsWith('$');++ if (anchored) p = p.slice(0, -1);++ const body = p.split('*').map(s => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');++ return new RegExp('^' + body + (anchored ? '$' : ''));++}++++function normalizePath(p) {++ // Compare percent-encoding consistently: decode what is safe to decode, re-encode the rest.++ try { return encodeURI(decodeURI(p)); } catch { return p; }++}++++/** Is `pathWithQuery` allowed by the parsed rules? */++function isAllowed(parsed, pathWithQuery) {++ const target = normalizePath(pathWithQuery || '/');++ let best = null;++ for (const r of parsed.rules) {++ const re = ruleRegex(normalizePath(r.path));++ if (!re.test(target)) continue;++ const len = r.path.length;++ if (!best || len > best.len || (len === best.len && r.allow && !best.allow)) best = { len, allow: r.allow };++ }++ return best ? best.allow : true;++}++++/**++ * createRobots({ db, fetcher, agent, ttlMs, maxBytes, now, beforeRequest(host) })++ * check(url) → { allowed, reason, crawlDelaySec, source: 'cache'|'fetched' }++ */++function createRobots({ db, fetcher, agent, ttlMs, maxBytes, now = () => Date.now(), beforeRequest = async () => {} }) {++ const get = db.prepare('SELECT * FROM robots_cache WHERE origin = ?');++ const put = db.prepare(`INSERT INTO robots_cache (origin, fetched_at, expires_at, status, outcome, rules, crawl_delay_sec, detail)++ VALUES (@origin, @fetched_at, @expires_at, @status, @outcome, @rules, @crawl_delay_sec, @detail)++ ON CONFLICT(origin) DO UPDATE SET fetched_at = excluded.fetched_at, expires_at = excluded.expires_at, status = excluded.status,++ outcome = excluded.outcome, rules = excluded.rules, crawl_delay_sec = excluded.crawl_delay_sec, detail = excluded.detail`);++ const inflight = new Map();++++ async function load(origin) {++ await beforeRequest(new URL(origin).host);++ const t = now();++ let row;++ try {++ const r = await fetcher.fetchUrl(`${origin}/robots.txt`, { headers: { Accept: 'text/plain' }, capBytes: maxBytes });++ if (r.status >= 200 && r.status < 300) {++ const parsed = parseRobots(r.body.toString('utf8'), agent);++ row = { status: r.status, outcome: 'parsed', rules: JSON.stringify(parsed.rules), crawl_delay_sec: parsed.crawlDelaySec, detail: null };++ } else if (r.status >= 400 && r.status < 500 && r.status !== 429) {++ row = { status: r.status, outcome: 'unavailable', rules: '[]', crawl_delay_sec: null, detail: `HTTP ${r.status}: no restrictions` };++ } else {++ row = { status: r.status, outcome: 'unreachable', rules: '[]', crawl_delay_sec: null, detail: `HTTP ${r.status}: complete disallow` };++ }++ } catch (err) {++ // Network error, timeout, refused address or an oversized file: refuse, never guess.++ row = { status: null, outcome: 'unreachable', rules: '[]', crawl_delay_sec: null, detail: `${err.code || 'error'}: ${err.message}`.slice(0, 300) };++ }++ // An unreachable robots.txt is retried sooner than a parsed one.++ const ttl = row.outcome === 'unreachable' ? Math.min(ttlMs, 15 * 60 * 1000) : ttlMs;++ const full = { origin, fetched_at: t, expires_at: t + ttl, ...row };++ put.run(full);++ return full;++ }++++ async function entry(origin) {++ const cached = get.get(origin);++ if (cached && cached.expires_at > now()) return { row: cached, source: 'cache' };++ if (!inflight.has(origin)) inflight.set(origin, load(origin).finally(() => inflight.delete(origin)));++ return { row: await inflight.get(origin), source: 'fetched' };++ }++++ async function check(input) {++ const url = new URL(input);++ const { row, source } = await entry(url.origin);++ if (row.outcome === 'unreachable') {++ // A refused address/port is a policy refusal of the host itself, not a robots answer.++ const code = row.status == null ? String(row.detail || '').split(':')[0] : null;++ return { allowed: false, reason: `robots.txt unreachable (${row.detail})`, crawlDelaySec: null, source, refusedCode: ['address_refused', 'port_refused', 'bad_url'].includes(code) ? code : null };++ }++ if (row.outcome === 'unavailable') return { allowed: true, reason: null, crawlDelaySec: null, source };++ const parsed = { rules: JSON.parse(row.rules) };++ const allowed = isAllowed(parsed, url.pathname + url.search);++ return { allowed, reason: allowed ? null : `disallowed by ${url.origin}/robots.txt`, crawlDelaySec: row.crawl_delay_sec, source };++ }++++ return { check, forget: (origin) => db.prepare('DELETE FROM robots_cache WHERE origin = ?').run(origin) };++}++++module.exports = { parseRobots, isAllowed, createRobots };nnew file mode 100644@@@ -0,0 +1,43 @@++'use strict';++/**++ * In-process scheduler: every SOURCES_TICK_MS, start runs for enabled, fetched sources whose++ * next_due_at and not_before have passed, oldest first, up to SOURCES_MAX_CONCURRENT runs at a++ * time. ingest.run() itself refuses a second run of a source already in flight.++ */++function createScheduler({ db, ingest, config, now = () => Date.now(), log = console }) {++ const due = db.prepare(`SELECT key FROM sources WHERE enabled = 1 AND type != 'manual'++ AND next_due_at <= @now AND not_before <= @now ORDER BY next_due_at, key LIMIT @limit`);++ let timer = null;++ const running = new Set();++++ function tick() {++ const free = config.worker.maxConcurrent - running.size;++ if (free <= 0) return [];++ const keys = due.all({ now: now(), limit: free + running.size }).map(r => r.key).filter(k => !running.has(k)).slice(0, free);++ for (const key of keys) {++ running.add(key);++ ingest.run(key, { trigger: 'schedule' })++ .catch(err => log.error(`[scheduler] ${key}: ${err.stack || err}`))++ .finally(() => running.delete(key));++ }++ return keys;++ }++++ function start() {++ if (timer) return;++ timer = setInterval(() => { try { tick(); } catch (err) { log.error(`[scheduler] tick: ${err.message}`); } }, config.worker.tickMs);++ timer.unref?.();++ }++++ async function stop() {++ if (timer) clearInterval(timer);++ timer = null;++ // let runs in flight finish (each is bounded by the fetch timeout)++ const deadline = Date.now() + 15000;++ while (running.size && Date.now() < deadline) await new Promise(r => setTimeout(r, 50));++ }++++ return { start, stop, tick, running: () => Boolean(timer), active: () => [...running] };++}++++module.exports = { createScheduler };nnew file mode 100644@@@ -0,0 +1,38 @@++'use strict';++/**++ * Per-host request spacing shared by robots.txt fetches and ingestion: before any request to a++ * host, wait until max(SOURCES_HOST_MIN_INTERVAL_MS, the source's min interval, the host's robots++ * Crawl-delay) has passed since the previous request to it. Requests to one host are serialized.++ */++const sleep = (ms) => new Promise(r => setTimeout(r, ms));++++function createSpacer({ hostMinIntervalMs = 1000, now = () => Date.now() } = {}) {++ const hosts = new Map(); // host → { last, crawlDelayMs, chain }++++ function state(host) {++ let h = hosts.get(host);++ if (!h) { h = { last: 0, crawlDelayMs: 0, chain: Promise.resolve() }; hosts.set(host, h); }++ return h;++ }++++ function space(host, minMs = 0) {++ const h = state(host);++ const turn = h.chain.then(async () => {++ const gap = Math.max(hostMinIntervalMs, minMs || 0, h.crawlDelayMs);++ const wait = h.last + gap - now();++ if (wait > 0) await sleep(wait);++ h.last = now();++ });++ h.chain = turn.catch(() => {});++ return turn;++ }++++ function setCrawlDelay(host, sec) {++ if (sec == null) return;++ state(host).crawlDelayMs = Math.min(sec * 1000, 3600 * 1000);++ }++++ return { space, setCrawlDelay };++}++++module.exports = { createSpacer };nnew file mode 100644@@@ -0,0 +1,166 @@++'use strict';++/** Parsers against fixtures: what the source states is kept, what it does not state stays null. */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const feed = require('../server/adapters/feed');++const sitemap = require('../server/adapters/sitemap');++const jsonld = require('../server/adapters/jsonld');++const api = require('../server/adapters/api');++const { decodeBody, toIsoDate, canonicalUrl } = require('../server/adapters/util');++const { suite } = require('./helpers');++++const t = suite('adapters');++const fx = (name) => fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');++++t('RSS 2.0: guid identity, cleaned link, text-only summary, dates, skips', () => {++ const r = feed.parse(fx('rss2.xml'), { url: 'https://news.example.org/feed.xml' });++ assert.strictEqual(r.meta.format, 'rss2');++ assert.strictEqual(r.items.length, 2);++ assert.strictEqual(r.skipped, 2, 'the identity-less item and the duplicate guid');++ const [a, b] = r.items;++ assert.strictEqual(a.identity, 'news-0001');++ assert.strictEqual(a.canonical_url, 'https://news.example.org/2026/09/first-light?id=7', 'tracking params and fragment removed');++ assert.strictEqual(a.title, 'Telescope sees first light');++ assert.strictEqual(a.summary, 'The new telescope & its camera saw first light .');++ assert.ok(!a.summary.includes('alert'), 'scripts are dropped');++ assert.strictEqual(a.published_at, '2026-09-21T14:30:00.000Z');++ assert.deepStrictEqual(a.authors, ['Ada Example']);++ assert.deepStrictEqual(a.fields.categories, ['Astronomy', 'Instruments']);++ assert.deepStrictEqual(a.fields.enclosure, { url: 'https://news.example.org/img/first-light.jpg', type: 'image/jpeg', length: 12345 });++ assert.strictEqual(b.identity, 'https://news.example.org/2026/09/undated');++ assert.strictEqual(b.published_at, null, 'no date in the feed = no date, never "now"');++ assert.deepStrictEqual(b.authors, []);++});++++t('Atom: id identity, relative alternate link, html title, bad dates are null', () => {++ const r = feed.parse(fx('atom.xml'), { url: 'https://blog.example.com/feed.atom' });++ assert.strictEqual(r.meta.format, 'atom');++ assert.strictEqual(r.items.length, 2);++ const [a, b] = r.items;++ assert.strictEqual(a.identity, 'tag:blog.example.com,2026:posts/release-2');++ assert.strictEqual(a.canonical_url, 'https://blog.example.com/posts/release-2');++ assert.strictEqual(a.title, 'Release 2.0 notes');++ assert.strictEqual(a.published_at, '2026-09-19T10:00:00.000Z');++ assert.strictEqual(a.updated_at, '2026-09-20T18:30:02.000Z');++ assert.deepStrictEqual(a.authors, ['Grace Example']);++ assert.deepStrictEqual(a.fields.categories, ['releases']);++ assert.strictEqual(b.updated_at, null);++ assert.strictEqual(b.summary, 'Body');++});++++t('a large real-world-sized feed with thousands of entities parses (no expansion limit tripped)', () => {++ const items = Array.from({ length: 1500 }, (_, i) => `<item><title>Deal ${i} & more — $${i}</title><link>https://deals.example.com/d/${i}?a=1&b=2</link><guid>d-${i}</guid></item>`).join('');++ const r = feed.parse(`<?xml version="1.0"?><rss version="2.0"><channel><title>Big & busy</title>${items}</channel></rss>`, { url: 'https://deals.example.com/rss', maxItems: 2000 });++ assert.strictEqual(r.items.length, 1500);++ assert.strictEqual(r.items[7].title, 'Deal 7 & more — $7');++ assert.strictEqual(r.items[7].canonical_url, 'https://deals.example.com/d/7?a=1&b=2');++});++++t('RSS 1.0 (RDF)', () => {++ const r = feed.parse(fx('rss1.rdf'), { url: 'https://rdf.example.net/index.rdf' });++ assert.strictEqual(r.meta.format, 'rss1');++ assert.strictEqual(r.items[0].identity, 'https://rdf.example.net/a');++ assert.strictEqual(r.items[0].published_at, '2026-09-18T08:00:00.000Z');++});++++t('XML with entity declarations is refused; garbage is a parse error', () => {++ assert.throws(() => feed.parse(fx('xxe.xml'), { url: 'https://x.example/' }), (e) => e.code === 'parse_error' && /entity/.test(e.message));++ assert.throws(() => feed.parse('<html><body>not a feed</body></html>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');++ assert.throws(() => feed.parse('<rss><channel><item>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');++});++++t('sitemap urlset with the news extension; index lists children without following them', () => {++ const r = sitemap.parse(fx('sitemap.xml'), { url: 'https://deals.example.com/sitemap.xml' });++ assert.strictEqual(r.items.length, 2);++ assert.strictEqual(r.skipped, 1, 'a non-http loc');++ assert.strictEqual(r.items[0].canonical_url, 'https://deals.example.com/offers/1');++ assert.strictEqual(r.items[0].updated_at, '2026-09-21T00:00:00.000Z');++ assert.strictEqual(r.items[0].fields.priority, 0.8);++ assert.strictEqual(r.items[0].title, null);++ assert.strictEqual(r.items[1].title, 'Launch week');++ assert.strictEqual(r.items[1].published_at, '2026-09-20T09:00:00.000Z');++ assert.strictEqual(r.items[1].fields.publication, 'Deals Example');++ const idx = sitemap.parse(fx('sitemap-index.xml'), { url: 'https://deals.example.com/sitemap-index.xml' });++ assert.deepStrictEqual(idx.items.map(i => [i.kind, i.canonical_url]), [++ ['sitemap', 'https://deals.example.com/sitemap-1.xml'], ['sitemap', 'https://deals.example.com/sitemap-2.xml'],++ ]);++});++++t('JSON-LD Product: offers as stated, missing price stays null, rating copied not computed, review text not kept', () => {++ const r = jsonld.parse(fx('product.html'), { url: 'https://shop.example.com/p/widget-pro' });++ assert.strictEqual(r.items.length, 1, 'BreadcrumbList and Organization are not items');++ const p = r.items[0];++ assert.strictEqual(p.kind, 'product');++ assert.strictEqual(p.identity, 'https://shop.example.com/p/widget-pro');++ assert.strictEqual(p.title, 'Widget Pro');++ assert.strictEqual(p.summary, 'A very good widget.');++ assert.strictEqual(p.fields.sku, 'WP-100');++ assert.strictEqual(p.fields.gtin, '0012345678905');++ assert.strictEqual(p.fields.brand, 'Example Co');++ assert.deepStrictEqual(p.fields.offers[0], {++ type: 'Offer', price: '19.99', low_price: null, high_price: null, offer_count: null, currency: 'USD', availability: 'InStock',++ condition: null, valid_until: '2026-12-31T00:00:00.000Z', url: 'https://shop.example.com/p/widget-pro', seller: 'Example Shop',++ });++ assert.strictEqual(p.fields.offers[1].price, null, 'no price on the page = null, never 0 or a guess');++ assert.strictEqual(p.fields.offers[1].availability, 'OutOfStock');++ assert.deepStrictEqual(p.fields.aggregate_rating, { value: 4.4, best: null, worst: null, count: null, review_count: 89 });++ assert.deepStrictEqual(p.fields.reviews, [{ author: 'Sam', rating: { value: 5, best: 5, worst: null, count: null, review_count: null }, date: '2026-08-01T00:00:00.000Z' }]);++ assert.ok(!JSON.stringify(p).includes('Long text'), 'embedded review bodies are not stored');++});++++t('JSON-LD NewsArticle, Review and BlogPosting; an invalid block is skipped, not fatal', () => {++ const r = jsonld.parse(fx('article.html'), { url: 'https://times.example.org/2026/09/budget' });++ assert.deepStrictEqual(r.items.map(i => i.kind), ['article', 'review', 'article']);++ const [news, review, blog] = r.items;++ assert.strictEqual(news.title, 'Council approves budget');++ assert.strictEqual(news.canonical_url, 'https://times.example.org/2026/09/budget');++ assert.strictEqual(news.published_at, '2026-09-21T05:00:00.000Z');++ assert.deepStrictEqual(news.authors, ['Lee Reporter']);++ assert.strictEqual(news.fields.publisher, 'Example Times');++ assert.strictEqual(review.fields.item_reviewed, 'Widget Pro');++ assert.deepStrictEqual(review.fields.rating, { value: 3, best: 5, worst: null, count: null, review_count: null });++ assert.strictEqual(review.summary, 'Solid, not spectacular.');++ assert.strictEqual(blog.identity, 'https://times.example.org/blog/why-widgets');++ assert.strictEqual(r.skipped, 1, 'the invalid JSON block');++ assert.notStrictEqual(news.identity, review.identity);++});++++t('a page without JSON-LD is a parse error, not an empty success', () => {++ assert.throws(() => jsonld.parse(fx('no-jsonld.html'), { url: 'https://x.example/' }), (e) => e.code === 'parse_error');++ assert.throws(() => jsonld.parse('<script type="application/ld+json">nope</script>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');++});++++t('API mapping: fields by path, epoch dates, nulls kept, identity-less records skipped', () => {++ const endpoint = {++ format: 'json', items_path: 'deals', item_kind: 'offer',++ fields: { identity: 'dealID', url: 'link', title: 'title', published_at_epoch: 'releaseDate' },++ extra: { sale_price: 'salePrice', normal_price: 'normalPrice', store_id: 'storeID' },++ };++ assert.strictEqual(api.checkMapping(endpoint), null);++ const r = api.parse(fx('deals.json'), { url: 'https://api.example.com/deals', endpoint });++ assert.strictEqual(r.items.length, 2);++ assert.strictEqual(r.skipped, 1);++ assert.deepStrictEqual(r.items[0].fields, { sale_price: '4.99', normal_price: '19.99', store_id: '1' });++ assert.strictEqual(r.items[0].published_at, '2024-09-20T00:00:00.000Z');++ assert.strictEqual(r.items[1].fields.sale_price, null);++ assert.strictEqual(r.items[1].canonical_url, null);++ assert.strictEqual(r.items[1].published_at, null);++ assert.throws(() => api.parse('{"deals": {}}', { url: 'https://api.example.com/', endpoint }), (e) => e.code === 'parse_error');++ assert.throws(() => api.parse('not json', { url: 'https://api.example.com/', endpoint }), (e) => e.code === 'parse_error');++ assert.match(api.checkMapping({ fields: { title: 'a' } }), /identity/);++ assert.match(api.checkMapping({ fields: { identity: 'a b' } }), /malformed/);++});++++t('helpers: charset decoding, date and URL strictness', () => {++ const latin = Buffer.from('<?xml version="1.0" encoding="ISO-8859-1"?><x>caf\xe9</x>', 'latin1');++ assert.ok(decodeBody(latin, 'text/xml').includes('café'));++ assert.strictEqual(toIsoDate('1726790400'), null, 'bare numbers are ambiguous');++ assert.strictEqual(toIsoDate('yesterday'), null);++ assert.strictEqual(toIsoDate('1970-01-01'), null, 'implausible years are refused');++ assert.strictEqual(canonicalUrl('javascript:alert(1)'), null);++ assert.strictEqual(canonicalUrl('https://user:[email protected]/'), null);++ assert.strictEqual(canonicalUrl('HTTPS://Example.COM:443/a?gclid=1&b=2#x'), 'https://example.com/a?b=2');++});++++t.run();nnew file mode 100644@@@ -0,0 +1,155 @@++'use strict';++/** Read and admin API: capabilities, registry validation, items paging, manual items, removal, health. */++const assert = require('assert');++const { boot, request, serviceToken, site, sourceDef, rss, suite } = require('./helpers');++++const t = suite('api');++let svc;++let web;++const ADMIN = serviceToken('network', ['sources.source.manage']);++const READER = serviceToken('news', ['sources.source.read', 'sources.item.read']);++const ITEMS_ONLY = serviceToken('deals', ['sources.item.read']);++++const api = (method, p, opts = {}) => request(svc.base, method, p, opts);++++t('boot', async () => {++ svc = await boot();++ web = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/feed.xml': () => ({ body: rss([1, 2, 3, 4, 5].map(i => ({ guid: `g${i}`, title: `Item ${i}`, link: `https://example.org/${i}` }))) }),++ });++});++++t('every route needs a service token with its one capability', async () => {++ assert.strictEqual((await api('GET', '/api/v1/sources')).status, 401);++ assert.strictEqual((await api('GET', '/api/v1/sources', { token: ITEMS_ONLY })).status, 403);++ assert.strictEqual((await api('GET', '/api/v1/items', { token: serviceToken('news', ['sources.source.read']) })).status, 403);++ assert.strictEqual((await api('POST', '/api/v1/sources', { token: READER, body: sourceDef() })).status, 403);++ assert.strictEqual((await api('GET', '/api/v1/sources', { token: serviceToken('news', ['sources.source.read'], { aud: 'openvibe.search' }) })).status, 401);++ assert.strictEqual((await api('GET', '/api/v1/sources', { token: serviceToken('news', ['sources.*']) })).status, 200, 'a family grant covers it');++});++++t('registry validation: terms before enabling, credentials by SOURCES_CRED_ name only, sane types', async () => {++ const bad = async (def, re) => {++ const r = await api('POST', '/api/v1/sources', { token: ADMIN, body: def });++ assert.strictEqual(r.status, 422, JSON.stringify(def));++ assert.match(r.body.detail, re);++ };++ await bad(sourceDef({ terms_note: undefined, endpoints: ['https://example.org/feed'] }), /terms_note/);++ await bad(sourceDef({ robots_note: undefined, endpoints: ['https://example.org/feed'] }), /robots_note/);++ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'bearer', env: 'OV_OAUTH_CLIENT_SECRET' } }), /SOURCES_CRED_/);++ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'bearer', env: 'SOURCES_CRED_XX', value: 'secret' } }), /unknown auth field/);++ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'header', env: 'SOURCES_CRED_XX', header: 'Cookie' } }), /header/);++ await bad(sourceDef({ endpoints: ['file:///etc/passwd'] }), /http/);++ await bad(sourceDef({ endpoints: [] }), /endpoint/);++ await bad(sourceDef({ type: 'manual', endpoints: ['https://example.org/'] }), /manual/);++ await bad(sourceDef({ category: 'gossip', endpoints: ['https://example.org/'] }), /category/);++ await bad(sourceDef({ type: 'api', endpoints: [{ url: 'https://api.example.org/', fields: { title: 'x' } }] }), /identity/);++ await bad(sourceDef({ endpoints: ['https://example.org/'], sensitivity: 'financial', default_indexability: 'index', review_required: false }), /review/);++ // disabled without notes is fine: a proposal waiting for review++ const draft = await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'proposal', name: 'Proposal', type: 'rss', category: 'blog', endpoints: ['https://example.org/feed'] } });++ assert.strictEqual(draft.status, 201);++ assert.deepStrictEqual([draft.body.source.enabled, draft.body.source.review_required, draft.body.source.default_indexability], [false, true, 'noindex']);++ assert.strictEqual(draft.body.source.health.status, 'disabled');++ assert.strictEqual((await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'proposal', name: 'x', type: 'rss', category: 'blog', endpoints: ['https://example.org/f'] } })).status, 409);++ const enable = await api('PATCH', '/api/v1/sources/proposal', { token: ADMIN, body: { enabled: true } });++ assert.strictEqual(enable.status, 422);++ assert.strictEqual(enable.body.code, 'sources.terms_required');++});++++t('create, read, fetch, list runs; views never carry a credential value', async () => {++ const def = sourceDef({ key: 'feed-a', endpoints: [`${web.origin}/feed.xml`], auth: { mode: 'query', env: 'SOURCES_CRED_FEED_A', param: 'key' } });++ const c = await api('POST', '/api/v1/sources', { token: ADMIN, body: def });++ assert.strictEqual(c.status, 201);++ assert.deepStrictEqual(c.body.source.auth, { mode: 'query', env: 'SOURCES_CRED_FEED_A', param: 'key' });++ // the variable is not set in this process: the fetch is recorded as disabled++ let f = await api('POST', '/api/v1/sources/feed-a/fetch', { token: ADMIN });++ assert.strictEqual(f.body.runs[0].state, 'disabled');++ await api('PATCH', '/api/v1/sources/feed-a', { token: ADMIN, body: { auth: { mode: 'none' } } });++ svc.db.prepare('UPDATE sources SET last_request_at = NULL').run();++ f = await api('POST', '/api/v1/sources/feed-a/fetch', { token: ADMIN });++ assert.strictEqual(f.status, 200);++ assert.deepStrictEqual(f.body.runs.map(r => [r.state, r.items.created]), [['ok', 5]]);++ const runs = await api('GET', '/api/v1/sources/feed-a/runs', { token: READER });++ assert.deepStrictEqual(runs.body.runs.map(r => r.state), ['ok', 'disabled']);++ const all = await api('GET', '/api/v1/runs?state=failed', { token: READER });++ assert.strictEqual(all.status, 200);++ const one = await api('GET', '/api/v1/sources/feed-a', { token: READER });++ assert.strictEqual(one.body.source.health.status, 'healthy');++ assert.strictEqual((await api('GET', '/api/v1/sources/nope', { token: READER })).status, 404);++});++++t('items page in change order and carry provenance and source staleness', async () => {++ const p1 = await api('GET', '/api/v1/items?source=feed-a&limit=2', { token: ITEMS_ONLY });++ assert.strictEqual(p1.status, 200);++ assert.strictEqual(p1.body.items.length, 2);++ assert.strictEqual(p1.body.more, true);++ const p2 = await api('GET', `/api/v1/items?source=feed-a&limit=10&after=${p1.body.next_after}`, { token: ITEMS_ONLY });++ assert.strictEqual(p2.body.items.length, 3);++ assert.strictEqual(p2.body.more, false);++ const it = p1.body.items[0];++ assert.ok(it.provenance.retrieved_at && it.provenance.content_hash && it.provenance.raw_body_hash && it.provenance.parser_version === 'feed@1');++ assert.strictEqual(it.provenance.terms_note, 'test fixture');++ assert.deepStrictEqual(Object.keys(p1.body.sources), ['feed-a']);++ assert.strictEqual(p1.body.sources['feed-a'].stale, false);++ const byCat = await api('GET', '/api/v1/items?category=news', { token: ITEMS_ONLY });++ assert.strictEqual(byCat.body.items.length, 5);++ assert.strictEqual((await api('GET', '/api/v1/items?category=gossip', { token: ITEMS_ONLY })).status, 400);++ // resume after the last page: nothing new yet++ const tail = await api('GET', `/api/v1/items?source=feed-a&after=${p2.body.next_after}`, { token: ITEMS_ONLY });++ assert.deepStrictEqual([tail.body.items.length, tail.body.next_after], [0, p2.body.next_after]);++});++++t('removal needs a reason, is visible with include_removed, and appears in the change feed', async () => {++ const list = await api('GET', '/api/v1/items?source=feed-a', { token: ITEMS_ONLY });++ const target = list.body.items[0];++ const last = list.body.next_after;++ assert.strictEqual((await api('DELETE', `/api/v1/items/${target.id}`, { token: ADMIN, body: {} })).status, 422);++ const del = await api('DELETE', `/api/v1/items/${target.id}`, { token: ADMIN, body: { reason: 'licence withdrawn' } });++ assert.strictEqual(del.status, 200);++ assert.match(del.body.item.removed.reason, /licence withdrawn \(by svc:network\)/);++ const feed = await api('GET', `/api/v1/items?source=feed-a&after=${last}&include_removed=1`, { token: ITEMS_ONLY });++ assert.deepStrictEqual(feed.body.items.map(i => [i.id, Boolean(i.removed)]), [[target.id, true]]);++ const hidden = await api('GET', `/api/v1/items?source=feed-a&after=${last}`, { token: ITEMS_ONLY });++ assert.strictEqual(hidden.body.items.length, 0);++ const one = await api('GET', `/api/v1/items/${target.id}?revisions=1`, { token: ITEMS_ONLY });++ assert.deepStrictEqual(one.body.item.revisions.map(r => r.revision), [1, 2]);++ assert.strictEqual((await api('DELETE', '/api/v1/sources/feed-a', { token: ADMIN })).status, 409, 'a source with items cannot be deleted');++ const gone = await api('DELETE', '/api/v1/sources/proposal', { token: ADMIN });++ assert.strictEqual(gone.status, 204, gone.text + JSON.stringify(svc.db.prepare('SELECT key FROM sources').all()));++});++++t('manual sources: items need an evidence URL; fetches are refused', async () => {++ const m = await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'staff-coupons', name: 'Staff-entered codes', type: 'manual', category: 'coupons', terms_note: 'codes the merchant published itself', enabled: true } });++ assert.strictEqual(m.status, 201);++ assert.strictEqual(m.body.source.health.status, 'manual');++ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/fetch', { token: ADMIN })).status, 409);++ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10' } })).status, 422);++ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10', url: 'https://merchant.example/promo', published_at: 'soon' } })).status, 422);++ const add = await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: {++ title: 'SAVE10', url: 'https://merchant.example/promo?utm_source=x', kind: 'coupon', fields: { code: 'SAVE10', expires: null },++ } });++ assert.strictEqual(add.status, 201);++ assert.strictEqual(add.body.item.canonical_url, 'https://merchant.example/promo');++ assert.strictEqual(add.body.item.fields.expires, null, 'unknown expiry stays unknown');++ assert.strictEqual(add.body.item.provenance.entered_by, 'svc:network');++ assert.strictEqual(add.body.item.provenance.parser_version, 'manual@1');++ const again = await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10', url: 'https://merchant.example/promo', kind: 'coupon', fields: { code: 'SAVE10', expires: null } } });++ assert.strictEqual(again.body.outcome, 'unchanged');++ assert.strictEqual((await api('POST', '/api/v1/sources/feed-a/items', { token: ADMIN, body: { title: 'x', url: 'https://x.example/' } })).status, 409);++});++++t('health lists what needs attention', async () => {++ await api('POST', '/api/v1/sources', { token: ADMIN, body: sourceDef({ key: 'never', endpoints: ['https://example.org/feed'] }) });++ const h = await api('GET', '/api/v1/health', { token: READER });++ assert.strictEqual(h.status, 200);++ assert.ok(h.body.counts.healthy >= 1 && h.body.counts.manual === 1);++ assert.ok(h.body.attention.some(a => a.key === 'never' && a.status === 'never_fetched'));++ const ready = await api('GET', '/api/ready');++ assert.strictEqual(ready.status, 200);++ assert.strictEqual(ready.body.checks.db, true);++});++++t('done', async () => { await web.close(); await svc.stop(); });++++t.run();nnew file mode 100644@@@ -0,0 +1,10 @@++<html><head>++<script type="application/ld+json">++[++ { "@context": "https://schema.org", "@type": "NewsArticle", "headline": "Council approves budget", "datePublished": "2026-09-21T07:00:00+02:00", "dateModified": "2026-09-21T09:00:00+02:00", "author": [{ "@type": "Person", "name": "Lee Reporter" }], "publisher": { "@type": "Organization", "name": "Example Times" }, "description": "The council voted 7-2." },++ { "@context": "https://schema.org", "@type": "Review", "name": "Review of Widget Pro", "itemReviewed": { "@type": "Product", "name": "Widget Pro" }, "author": { "@type": "Person", "name": "Critic" }, "reviewRating": { "@type": "Rating", "ratingValue": "3", "bestRating": "5" }, "datePublished": "2026-09-10", "reviewBody": "Solid, not spectacular." },++ { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Why widgets", "url": "https://times.example.org/blog/why-widgets", "datePublished": "2026-09-01" }++]++</script>++<script type="application/ld+json">{ this is not json }</script>++</head></html>nnew file mode 100644@@@ -0,0 +1,25 @@++<?xml version="1.0" encoding="utf-8"?>++<feed xmlns="http://www.w3.org/2005/Atom">++ <title>Example Engineering Blog</title>++ <link href="https://blog.example.com/"/>++ <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>++ <updated>2026-09-20T18:30:02Z</updated>++ <entry>++ <title type="html">Release <em>2.0</em> notes</title>++ <link rel="alternate" href="/posts/release-2"/>++ <link rel="enclosure" href="https://blog.example.com/files/notes.pdf"/>++ <id>tag:blog.example.com,2026:posts/release-2</id>++ <published>2026-09-19T10:00:00Z</published>++ <updated>2026-09-20T18:30:02Z</updated>++ <author><name>Grace Example</name></author>++ <category term="releases"/>++ <summary>What changed in 2.0.</summary>++ </entry>++ <entry>++ <title>Second post</title>++ <link href="https://blog.example.com/posts/second"/>++ <id>tag:blog.example.com,2026:posts/second</id>++ <updated>not a date</updated>++ <content type="html"><p>Body</p></content>++ </entry>++</feed>nnew file mode 100644@@@ -0,0 +1,7 @@++{++ "deals": [++ { "dealID": "d-1", "title": "Game One", "salePrice": "4.99", "normalPrice": "19.99", "storeID": "1", "releaseDate": 1726790400, "link": "https://deals.example.com/d/1" },++ { "dealID": "d-2", "title": "Game Two", "salePrice": null, "normalPrice": "9.99", "storeID": "2" },++ { "title": "No identity, skipped" }++ ]++}nnew file mode 100644@@@ -0,0 +1 @@++<html><head><title>Nothing structured</title></head><body>Hello</body></html>nnew file mode 100644@@@ -0,0 +1,30 @@++<!doctype html>++<html><head><meta charset="utf-8"><title>Widget Pro</title>++<script type="application/ld+json">++{++ "@context": "https://schema.org",++ "@graph": [++ { "@type": "BreadcrumbList", "itemListElement": [] },++ {++ "@type": "Product",++ "@id": "https://shop.example.com/p/widget-pro#product",++ "name": "Widget Pro",++ "sku": "WP-100",++ "gtin13": "0012345678905",++ "brand": { "@type": "Brand", "name": "Example Co" },++ "description": "A <b>very</b> good widget.",++ "image": ["https://shop.example.com/img/wp.jpg"],++ "offers": [++ { "@type": "Offer", "price": "19.99", "priceCurrency": "USD", "availability": "https://schema.org/InStock", "priceValidUntil": "2026-12-31", "url": "https://shop.example.com/p/widget-pro", "seller": { "@type": "Organization", "name": "Example Shop" } },++ { "@type": "Offer", "priceCurrency": "EUR", "availability": "https://schema.org/OutOfStock" }++ ],++ "aggregateRating": { "@type": "AggregateRating", "ratingValue": "4.4", "reviewCount": "89" },++ "review": [++ { "@type": "Review", "author": { "@type": "Person", "name": "Sam" }, "reviewRating": { "@type": "Rating", "ratingValue": 5, "bestRating": 5 }, "datePublished": "2026-08-01", "reviewBody": "Long text that we do not keep." }++ ]++ }++ ]++}++</script>++<script type="application/ld+json">{ "@context": "https://schema.org", "@type": "Organization", "name": "Example Shop" }</script>++</head><body>Widget</body></html>nnew file mode 100644@@@ -0,0 +1,13 @@++<?xml version="1.0"?>++<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://purl.org/rss/1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/">++ <channel rdf:about="https://rdf.example.net/">++ <title>RDF Site</title>++ <link>https://rdf.example.net/</link>++ </channel>++ <item rdf:about="https://rdf.example.net/a">++ <title>RDF item A</title>++ <link>https://rdf.example.net/a</link>++ <dc:date>2026-09-18T08:00:00Z</dc:date>++ <description>First.</description>++ </item>++</rdf:RDF>nnew file mode 100644@@@ -0,0 +1,32 @@++<?xml version="1.0" encoding="UTF-8"?>++<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">++ <channel>++ <title>Example Observatory News</title>++ <link>https://news.example.org/</link>++ <description>Fixture feed</description>++ <item>++ <title>Telescope sees first light</title>++ <link>https://news.example.org/2026/09/first-light?utm_source=rss&id=7#top</link>++ <guid isPermaLink="false">news-0001</guid>++ <pubDate>Mon, 21 Sep 2026 14:30:00 +0000</pubDate>++ <dc:creator>Ada Example</dc:creator>++ <category>Astronomy</category>++ <category>Instruments</category>++ <description><![CDATA[<p>The <b>new</b> telescope & its camera saw <a href="x">first light</a>.</p><script>alert(1)</script>]]></description>++ <enclosure url="https://news.example.org/img/first-light.jpg" type="image/jpeg" length="12345"/>++ </item>++ <item>++ <title>Undated item keeps no date</title>++ <link>https://news.example.org/2026/09/undated</link>++ <description>Plain text description.</description>++ </item>++ <item>++ <description>An item with neither guid nor link is skipped.</description>++ </item>++ <item>++ <title>Duplicate guid</title>++ <link>https://news.example.org/dup</link>++ <guid isPermaLink="false">news-0001</guid>++ </item>++ </channel>++</rss>nnew file mode 100644@@@ -0,0 +1,5 @@++<?xml version="1.0" encoding="UTF-8"?>++<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">++ <sitemap><loc>https://deals.example.com/sitemap-1.xml</loc><lastmod>2026-09-01T00:00:00Z</lastmod></sitemap>++ <sitemap><loc>https://deals.example.com/sitemap-2.xml</loc></sitemap>++</sitemapindex>nnew file mode 100644@@@ -0,0 +1,18 @@++<?xml version="1.0" encoding="UTF-8"?>++<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9">++ <url>++ <loc>https://deals.example.com/offers/1</loc>++ <lastmod>2026-09-21</lastmod>++ <changefreq>daily</changefreq>++ <priority>0.8</priority>++ </url>++ <url>++ <loc>https://deals.example.com/news/launch</loc>++ <news:news>++ <news:publication><news:name>Deals Example</news:name><news:language>en</news:language></news:publication>++ <news:publication_date>2026-09-20T09:00:00Z</news:publication_date>++ <news:title>Launch week</news:title>++ </news:news>++ </url>++ <url><loc>ftp://deals.example.com/not-http</loc></url>++</urlset>nnew file mode 100644@@@ -0,0 +1,3 @@++<?xml version="1.0"?>++<!DOCTYPE lolz [<!ENTITY lol "lol"><!ENTITY lol2 "&lol;&lol;&lol;&lol;">]>++<rss version="2.0"><channel><title>&lol2;</title><item><title>x</title><link>https://x.example/</link></item></channel></rss>nnew file mode 100644@@@ -0,0 +1,141 @@++'use strict';++/**++ * Shared test fixtures: a generated Network signing key, service token minting, a booted Sources++ * service on a random port with a temp database, and stub HTTP servers standing in for the++ * sites being ingested. Nothing here touches the internet.++ */++const crypto = require('crypto');++const fs = require('fs');++const os = require('os');++const path = require('path');++const nodeHttp = require('http');++const { serviceAuth } = require('openvibe-contracts');++const { load } = require('../server/config');++const { start } = require('../server/index');++++const ISSUER = 'https://openvibe.network';++const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', {++ modulusLength: 2048,++ privateKeyEncoding: { type: 'pkcs8', format: 'pem' },++ publicKeyEncoding: { type: 'spki', format: 'pem' },++});++++const silent = { log() {}, warn() {}, error(...a) { if (process.env.DEBUG) console.error(...a); } };++++function serviceToken(slug, cap, { aud = 'openvibe.sources', exp = Math.floor(Date.now() / 1000) + 300, key = privateKey } = {}) {++ const now = Math.floor(Date.now() / 1000);++ return serviceAuth.signServiceToken({++ iss: ISSUER, sub: `svc:${slug}`, actor_type: 'service', aud: [aud], cap, ns: [], iat: now, exp,++ jti: `tok_${crypto.randomBytes(8).toString('hex')}`,++ }, key);++}++++const made = [];++process.on('exit', () => { for (const d of made) fs.rmSync(d, { recursive: true, force: true }); });++++function tmpDir() {++ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-sources-test-'));++ made.push(d);++ return d;++}++++/** Boot Sources; the scheduler is off unless worker: 'on'. Loopback is allowlisted for the stubs. */++async function boot({ env = {}, worker = 'off', lookupImpl, tokenClient } = {}) {++ const dir = tmpDir();++ const config = load({++ NODE_ENV: 'test',++ PORT: '0',++ SOURCES_DB_PATH: path.join(dir, 'sources.db'),++ OV_NETWORK_PUBLIC_KEY: publicKey,++ SOURCES_WORKER: worker,++ SOURCES_ALLOW_PRIVATE_HOSTS: '127.0.0.1',++ SOURCES_HOST_MIN_INTERVAL_MS: '0',++ SOURCES_FETCH_TIMEOUT_MS: '1500',++ SOURCES_TICK_MS: '50',++ ...env,++ });++ const h = await start({ config, log: silent, lookupImpl, tokenClient });++ const base = `http://127.0.0.1:${h.server.address().port}`;++ return { ...h, base, dir, async stop() { await h.close(); } };++}++++async function request(base, method, p, { token, body, headers = {} } = {}) {++ const h = { ...headers };++ if (token) h.Authorization = `Bearer ${token}`;++ if (body !== undefined) h['Content-Type'] = 'application/json';++ const res = await fetch(base + p, { method, headers: h, body: body !== undefined ? JSON.stringify(body) : undefined });++ const text = await res.text();++ let json = null;++ try { json = text ? JSON.parse(text) : null; } catch { json = null; }++ return { status: res.status, body: json, text, headers: res.headers };++}++++/**++ * A stub website. routes: { '/path': (req, res, ctx) => void | { status, headers, body, delayMs } }.++ * Every request is recorded with its headers and arrival time.++ */++async function site(routes = {}) {++ const requests = [];++ const server = nodeHttp.createServer(async (req, res) => {++ const url = new URL(req.url, 'http://stub');++ requests.push({ path: url.pathname, search: url.search, headers: req.headers, at: Date.now() });++ const handler = routes[url.pathname] || routes['*'];++ if (!handler) { res.statusCode = 404; res.end('not found'); return; }++ const out = await handler(req, res, { url, count: requests.filter(r => r.path === url.pathname).length });++ if (!out || res.writableEnded) return;++ if (out.delayMs) await new Promise(r => setTimeout(r, out.delayMs));++ if (res.destroyed) return;++ res.writeHead(out.status || 200, out.headers || {});++ res.end(out.body === undefined ? '' : out.body);++ });++ await new Promise(r => server.listen(0, '127.0.0.1', r));++ const origin = `http://127.0.0.1:${server.address().port}`;++ return {++ origin,++ requests,++ hits: (p) => requests.filter(r => r.path === p),++ routes,++ close: () => new Promise(r => { server.closeAllConnections?.(); server.close(() => r()); }),++ };++}++++/** A minimal valid source definition pointing at a stub. */++function sourceDef(overrides = {}) {++ return {++ key: 'src-' + crypto.randomBytes(4).toString('hex'),++ name: 'Stub source',++ type: 'rss',++ category: 'news',++ endpoints: [],++ robots_note: 'robots.txt checked by the fetcher on every run',++ terms_note: 'test fixture',++ min_interval_ms: 0,++ poll_interval_sec: 3600,++ enabled: true,++ ...overrides,++ };++}++++function rss(items) {++ return `<?xml version="1.0"?><rss version="2.0"><channel><title>Stub</title>${items.map(i =>++ `<item><title>${i.title}</title><link>${i.link}</link><guid isPermaLink="false">${i.guid}</guid>${i.date ? `<pubDate>${i.date}</pubDate>` : ''}<description>${i.description || ''}</description></item>`).join('')}</channel></rss>`;++}++++const fixture = (name) => fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');++const sleep = (ms) => new Promise(r => setTimeout(r, ms));++++function suite(name) {++ const tests = [];++ const t = (n, fn) => tests.push([n, fn]);++ t.run = async () => {++ let failed = 0;++ for (const [n, fn] of tests) {++ try { await fn(); console.log(` ok ${n}`); } catch (err) { failed++; console.log(` FAIL ${n}\n${err.stack}`); }++ }++ console.log(`${name}: ${tests.length - failed}/${tests.length} passed`);++ if (failed) process.exit(1);++ };++ return t;++}++++module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, tmpDir, boot, request, site, sourceDef, rss, fixture, sleep, suite };nnew file mode 100644@@@ -0,0 +1,337 @@++'use strict';++/**++ * Ingestion against stub sites: explicit run states, conditional GET, failure never fabricates or++ * modifies items, staleness, rate limits, one run in flight, credentials by variable name.++ */++const assert = require('assert');++const { boot, site, sourceDef, rss, fixture, sleep, suite } = require('./helpers');++++const t = suite('ingest');++let svc;++let web;++let feedBody;++let feedStatus;++let feedEtag;++++const FEED_V1 = rss([++ { guid: 'a-1', title: 'Alpha', link: 'https://news.example.org/a', date: 'Mon, 21 Sep 2026 10:00:00 +0000', description: 'first' },++ { guid: 'b-1', title: 'Beta', link: 'https://news.example.org/b' },++]);++++function create(def) {++ svc.registry.create(def, 'test');++ return def.key;++}++const run = (key, trigger = 'manual') => svc.ingest.run(key, { trigger });++const itemsOf = (key) => svc.db.prepare('SELECT * FROM items WHERE source_key = ? ORDER BY identity').all(key);++const runsOf = (key) => svc.db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? ORDER BY rid').all(key);++const events = (type) => svc.outbox.all().filter(e => e.event_type === type);++const reset = (key) => svc.db.prepare('UPDATE sources SET last_request_at = NULL, not_before = 0 WHERE key = ?').run(key);++++t('boot', async () => {++ svc = await boot();++ feedBody = FEED_V1;++ feedStatus = 200;++ feedEtag = '"v1"';++ web = await site({++ '/robots.txt': () => ({ body: 'User-agent: *\nDisallow: /private/\n\nUser-agent: OpenVibeSources\nDisallow: /blocked-for-us\nAllow: /\n' }),++ '/feed.xml': (req) => {++ if (feedStatus !== 200) return { status: feedStatus, body: 'nope' };++ if (feedEtag && req.headers['if-none-match'] === feedEtag) return { status: 304 };++ return { headers: { 'Content-Type': 'application/rss+xml; charset=utf-8', ETag: feedEtag }, body: feedBody };++ },++ });++});++++t('a good fetch creates items with full provenance and an ok run', async () => {++ const key = create(sourceDef({ key: 'news-a', endpoints: [`${web.origin}/feed.xml`], license_note: 'headlines only', terms_note: 'fixture terms' }));++ const out = await run(key);++ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok']);++ const items = itemsOf(key);++ assert.strictEqual(items.length, 2);++ const a = items[0];++ assert.strictEqual(a.identity, 'a-1');++ assert.strictEqual(a.canonical_url, 'https://news.example.org/a');++ assert.strictEqual(a.revision, 1);++ assert.match(a.content_hash, /^[0-9a-f]{64}$/);++ assert.match(a.raw_body_hash, /^[0-9a-f]{64}$/);++ assert.strictEqual(a.parser_version, 'feed@1');++ assert.strictEqual(a.license_note, 'headlines only');++ assert.strictEqual(a.terms_note, 'fixture terms');++ assert.ok(a.retrieved_at > 0 && a.first_seen_at === a.retrieved_at);++ assert.strictEqual(a.published_at, '2026-09-21T10:00:00.000Z');++ assert.strictEqual(items[1].published_at, null);++ const [r] = runsOf(key);++ assert.deepStrictEqual([r.state, r.http_status, r.items_created, r.items_seen], ['ok', 200, 2, 2]);++ assert.strictEqual(r.raw_body_hash, a.raw_body_hash);++ assert.strictEqual(events('sources.item.created').filter(e => e.payload.source_key === key).length, 2);++ // the fetcher identified itself and asked robots.txt first++ const feedReq = web.hits('/feed.xml')[0];++ assert.match(feedReq.headers['user-agent'], /^OpenVibeSources\//);++ assert.ok(web.hits('/robots.txt')[0].at <= feedReq.at);++});++++t('conditional GET: the stored ETag is sent and a 304 is not_modified with nothing changed', async () => {++ reset('news-a');++ const before = itemsOf('news-a');++ const out = await run('news-a');++ assert.deepStrictEqual(out.runs.map(r => r.state), ['not_modified']);++ const last = web.hits('/feed.xml').at(-1);++ assert.strictEqual(last.headers['if-none-match'], '"v1"');++ assert.deepStrictEqual(itemsOf('news-a').map(i => [i.revision, i.content_hash, i.retrieved_at]), before.map(i => [i.revision, i.content_hash, i.retrieved_at]));++});++++t('Last-Modified is used when there is no ETag', async () => {++ const lm = 'Mon, 21 Sep 2026 10:00:00 GMT';++ const s = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/f': (req) => (req.headers['if-modified-since'] === lm ? { status: 304 } : { headers: { 'Last-Modified': lm }, body: FEED_V1 }),++ });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/f`] }));++ assert.strictEqual((await run(key)).runs[0].state, 'ok');++ reset(key);++ assert.strictEqual((await run(key)).runs[0].state, 'not_modified');++ } finally { await s.close(); }++});++++t('a changed item is a new revision; the old one stays in item_revisions', async () => {++ reset('news-a');++ feedEtag = '"v2"';++ feedBody = rss([++ { guid: 'a-1', title: 'Alpha (corrected)', link: 'https://news.example.org/a', date: 'Mon, 21 Sep 2026 10:00:00 +0000', description: 'first, corrected' },++ { guid: 'b-1', title: 'Beta', link: 'https://news.example.org/b' },++ ]);++ const out = await run('news-a');++ assert.strictEqual(out.runs[0].state, 'ok');++ assert.deepStrictEqual([out.runs[0].items_updated, out.runs[0].items_unchanged], [1, 1]);++ const a = itemsOf('news-a')[0];++ assert.strictEqual(a.revision, 2);++ assert.strictEqual(a.title, 'Alpha (corrected)');++ const revs = svc.db.prepare('SELECT revision, snapshot FROM item_revisions WHERE item_id = ? ORDER BY revision').all(a.id);++ assert.deepStrictEqual(revs.map(r => [r.revision, JSON.parse(r.snapshot).title]), [[1, 'Alpha'], [2, 'Alpha (corrected)']]);++ const upd = events('sources.item.updated').find(e => e.payload.item_id === a.id);++ assert.ok(upd && upd.payload.previous_content_hash && upd.payload.revision === 2);++});++++t('failure never fabricates: http errors, timeouts, parse errors and oversize bodies leave items exactly as they were', async () => {++ const snapshot = () => JSON.stringify(itemsOf('news-a'));++ const before = snapshot();++ const count = svc.db.prepare('SELECT COUNT(*) AS n FROM items').get().n;++++ feedStatus = 500;++ reset('news-a');++ let out = await run('news-a');++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].http_status, out.runs[0].error_code], ['http_error', 500, 'http_500']);++++ feedStatus = 200;++ feedEtag = '"v3"';++ feedBody = '<rss><channel><item><title>Broken';++ reset('news-a');++ out = await run('news-a');++ assert.strictEqual(out.runs[0].state, 'parse_error');++ assert.match(out.runs[0].raw_body_hash, /^[0-9a-f]{64}$/, 'the unreadable body is identified by its hash');++++ feedBody = '<html><body>Service temporarily replaced by a login page</body></html>';++ reset('news-a');++ out = await run('news-a');++ assert.strictEqual(out.runs[0].state, 'parse_error');++++ assert.strictEqual(snapshot(), before, 'no item created, changed, re-dated or re-hashed');++ assert.strictEqual(svc.db.prepare('SELECT COUNT(*) AS n FROM items').get().n, count);++++ // after a failed parse the validator was not stored: the next good body is fetched in full++ feedBody = FEED_V1;++ reset('news-a');++ out = await run('news-a');++ assert.strictEqual(out.runs[0].state, 'ok');++ assert.notStrictEqual(web.hits('/feed.xml').at(-1).headers['if-none-match'], '"v3"');++++ const failed = events('sources.fetch.failed').filter(e => e.payload.source_key === 'news-a');++ assert.deepStrictEqual(failed.map(e => e.payload.state), ['http_error', 'parse_error', 'parse_error']);++ assert.deepStrictEqual(failed.map(e => e.payload.consecutive_failures), [1, 2, 3]);++ assert.strictEqual(svc.registry.get('news-a').consecutive_failures, 0, 'reset by the good run');++});++++t('timeouts and oversize bodies are explicit states', async () => {++ const s = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/slow': () => ({ delayMs: 3000, body: FEED_V1 }),++ '/huge': () => ({ body: 'x'.repeat(4096) }),++ });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/slow`] }));++ const out = await run(key);++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['timeout', 'timeout']);++ assert.strictEqual(itemsOf(key).length, 0);++++ const small = await boot({ env: { SOURCES_MAX_BYTES: '1024' } });++ try {++ small.registry.create(sourceDef({ key: 'huge', endpoints: [`${s.origin}/huge`] }), 'test');++ const r = await small.ingest.run('huge', { trigger: 'manual' });++ assert.deepStrictEqual([r.runs[0].state, r.runs[0].error_code], ['http_error', 'too_large']);++ } finally { await small.stop(); }++ } finally { await s.close(); }++});++++t('staleness is computed from the last success and exposed in health', async () => {++ const key = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], stale_after_sec: 1 }));++ let h = svc.registry.health(svc.registry.get(key));++ assert.deepStrictEqual([h.status, h.stale], ['never_fetched', true]);++ feedEtag = null;++ await run(key);++ h = svc.registry.health(svc.registry.get(key));++ assert.deepStrictEqual([h.status, h.stale], ['healthy', false]);++ await sleep(1100);++ h = svc.registry.health(svc.registry.get(key));++ assert.deepStrictEqual([h.status, h.stale], ['stale', true]);++ feedStatus = 503;++ reset(key);++ await run(key);++ h = svc.registry.health(svc.registry.get(key));++ assert.deepStrictEqual([h.status, h.stale, h.consecutive_failures], ['failing', true, 1]);++ feedStatus = 200;++});++++t('rate limit: requests to a source are spaced by min_interval_ms; an early trigger is rate_limited without a request', async () => {++ const s = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/one': () => ({ body: FEED_V1 }),++ '/two': () => ({ body: FEED_V1 }),++ });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/one`, `${s.origin}/two`], min_interval_ms: 400 }));++ const out = await run(key);++ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok', 'ok']);++ const gap = s.hits('/two')[0].at - s.hits('/one')[0].at;++ assert.ok(gap >= 380, `requests ${gap} ms apart`);++ const again = await run(key);++ assert.deepStrictEqual([again.runs[0].state, again.runs[0].error_code], ['rate_limited', 'local_rate_limit']);++ assert.strictEqual(s.hits('/one').length, 1, 'nothing was requested');++ assert.ok(!events('sources.fetch.failed').some(e => e.payload.source_key === key), 'our own limiter is not a source failure');++ } finally { await s.close(); }++});++++t('a 429 is rate_limited, stops the run and honours Retry-After', async () => {++ const s = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/a': () => ({ status: 429, headers: { 'Retry-After': '120' } }),++ '/b': () => ({ body: FEED_V1 }),++ });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/a`, `${s.origin}/b`] }));++ const out = await run(key);++ assert.deepStrictEqual(out.runs.map(r => [r.state, r.error_code]), [['rate_limited', 'upstream_429']]);++ assert.strictEqual(s.hits('/b').length, 0);++ const row = svc.registry.get(key);++ assert.ok(row.not_before - Date.now() > 100 * 1000, 'Retry-After respected');++ const early = await run(key);++ assert.strictEqual(early.runs[0].state, 'rate_limited');++ assert.strictEqual(s.hits('/a').length, 1);++ } finally { await s.close(); }++});++++t('one run per source in flight', async () => {++ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/slow': () => ({ delayMs: 300, body: FEED_V1 }) });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/slow`] }));++ const first = run(key);++ await sleep(50);++ const second = await run(key);++ assert.deepStrictEqual(second, { busy: true });++ assert.strictEqual((await first).runs[0].state, 'ok');++ assert.strictEqual(s.hits('/slow').length, 1);++ } finally { await s.close(); }++});++++t('disabled sources and missing credentials are recorded as disabled, nothing is requested', async () => {++ const key = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], enabled: false }));++ const n = web.requests.length;++ const out = await run(key);++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['disabled', 'source_disabled']);++ const key2 = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], auth: { mode: 'bearer', env: 'SOURCES_CRED_NOT_SET' } }));++ const out2 = await run(key2);++ assert.deepStrictEqual([out2.runs[0].state, out2.runs[0].error_code], ['disabled', 'credential_missing']);++ assert.match(out2.runs[0].detail, /SOURCES_CRED_NOT_SET/);++ assert.strictEqual(web.requests.length, n);++});++++t('a credential is read by name, sent only to its origin, and never stored', async () => {++ const other = await site({ '/robots.txt': () => ({ status: 404 }), '/landing': () => ({ body: FEED_V1 }) });++ const s = await site({++ '/robots.txt': () => ({ status: 404 }),++ '/api': () => ({ status: 302, headers: { Location: `${other.origin}/landing` } }),++ });++ const secret = 'sk_test_' + 'q'.repeat(24);++ const svc2 = await boot({ env: { SOURCES_CRED_EXAMPLE: secret } });++ try {++ svc2.registry.create(sourceDef({ key: 'cred', endpoints: [`${s.origin}/api`], auth: { mode: 'header', env: 'SOURCES_CRED_EXAMPLE', header: 'X-Api-Key' } }), 'test');++ const out = await svc2.ingest.run('cred', { trigger: 'manual' });++ assert.strictEqual(out.runs[0].state, 'ok');++ assert.strictEqual(s.hits('/api')[0].headers['x-api-key'], secret);++ assert.strictEqual(other.hits('/landing')[0].headers['x-api-key'], undefined, 'not forwarded across origins');++ const dump = JSON.stringify(svc2.db.prepare('SELECT * FROM sources').all()) + JSON.stringify(svc2.db.prepare('SELECT * FROM fetch_runs').all())++ + JSON.stringify(svc2.db.prepare('SELECT * FROM items').all()) + JSON.stringify(svc2.outbox.all());++ assert.ok(!dump.includes(secret), 'the secret value is nowhere in the database or events');++ assert.ok(dump.includes('SOURCES_CRED_EXAMPLE'), 'the variable name is recorded');++ } finally { await svc2.stop(); await s.close(); await other.close(); }++});++++t('removal is explicit and sticky; the next fetch does not bring the item back', async () => {++ const key = 'news-a';++ const a = itemsOf(key).find(i => i.identity === 'a-1');++ const source = svc.registry.fromRow(svc.registry.get(key));++ const removed = svc.items.remove(source, a.id, 'publisher takedown request', 'svc:network');++ assert.ok(removed.removed && /takedown/.test(removed.removed.reason));++ reset(key);++ feedEtag = '"v9"';++ feedBody = FEED_V1;++ await run(key);++ const again = svc.items.get(a.id);++ assert.ok(again.removed, 'still removed');++ assert.ok(events('sources.item.removed').some(e => e.payload.item_id === a.id));++});++++t('JSON-LD and sitemap sources ingest end to end', async () => {++ const s = await site({++ '/robots.txt': () => ({ body: 'User-agent: *\nAllow: /\n' }),++ '/p/widget-pro': () => ({ headers: { 'Content-Type': 'text/html; charset=utf-8' }, body: fixture('product.html') }),++ '/sitemap.xml': () => ({ headers: { 'Content-Type': 'application/xml' }, body: fixture('sitemap.xml') }),++ });++ try {++ const p = create(sourceDef({ type: 'jsonld', category: 'deals', endpoints: [`${s.origin}/p/widget-pro`] }));++ assert.strictEqual((await run(p)).runs[0].state, 'ok');++ const [prod] = itemsOf(p);++ assert.strictEqual(prod.kind, 'product');++ assert.strictEqual(JSON.parse(prod.fields).offers[0].price, '19.99');++ assert.strictEqual(prod.parser_version, 'jsonld@1');++ const sm = create(sourceDef({ type: 'sitemap', category: 'deals', endpoints: [`${s.origin}/sitemap.xml`] }));++ const out = await run(sm);++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].items_created, out.runs[0].items_skipped], ['ok', 2, 1]);++ } finally { await s.close(); }++});++++t('sources with search_visibility send staff-only, noindex index documents through the outbox', async () => {++ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: FEED_V1 }) });++ try {++ const key = create(sourceDef({ endpoints: [`${s.origin}/f`], search_visibility: 'members' }));++ await run(key);++ const docs = events('sources.index_document.upserted').filter(e => e.payload.facets.source === key);++ assert.strictEqual(docs.length, 2);++ const d = docs[0].payload;++ assert.deepStrictEqual([d.owner, d.type, d.visibility, d.indexability.decision], ['sources', 'item', 'members', 'noindex']);++ assert.deepStrictEqual(d.acl.groups, ['role:admin', 'role:global_mod']);++ assert.deepStrictEqual(docs[0].subject, { type: 'item', id: d.id, revision: d.revision });++ assert.strictEqual(docs[0].visibility, 'internal');++ const item = itemsOf(key)[0];++ svc.items.remove(svc.registry.fromRow(svc.registry.get(key)), item.id, 'test', null);++ const del = events('sources.index_document.deleted').find(e => e.payload.id === item.id);++ assert.deepStrictEqual(del.payload, { type: 'item', id: item.id, revision: 2 });++ // a source without search_visibility sends none++ assert.ok(!events('sources.index_document.upserted').some(e => e.payload.facets.source === 'news-a'));++ } finally { await s.close(); }++});++++t('done', async () => { await web.close(); await svc.stop(); });++++t.run();nnew file mode 100644@@@ -0,0 +1,80 @@++'use strict';++/**++ * Proposals for OpenVibe.Contracts are valid against the contracts' own schemas; what the API++ * returns matches the proposed sources.source@1 / sources.item@1; the seeds are honest.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const Ajv2020 = require('ajv/dist/2020');++const addFormats = require('ajv-formats');++const contracts = require('openvibe-contracts');++const { CAPS } = require('../server/auth');++const { validateSource, CATEGORIES } = require('../server/registry');++const { seed } = require('../scripts/seed');++const { boot, request, serviceToken, site, sourceDef, rss, suite } = require('./helpers');++++const t = suite('proposals');++const DOCS = path.join(__dirname, '..', 'docs');++const ajv = new Ajv2020({ allErrors: true, strict: false });++addFormats(ajv);++const schema = (name) => ajv.compile(JSON.parse(fs.readFileSync(path.join(DOCS, 'contracts-proposal', 'contracts', 'sources', name), 'utf8')));++const sourceSchema = schema('source.v1.json');++const itemSchema = schema('item.v1.json');++++t('capability proposals are valid capabilities.capability@1 and cover every enforced id', () => {++ const dir = path.join(DOCS, 'capabilities-proposal');++ const ids = fs.readdirSync(dir).filter(f => f.endsWith('.json')).map((f) => {++ const m = JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8'));++ const v = contracts.validate('capabilities.capability@1', m);++ assert.ok(v.valid, `${f}: ${JSON.stringify(v.errors)}`);++ assert.strictEqual(`${m.id}.json`, f);++ return m.id;++ });++ assert.deepStrictEqual(ids.sort(), Object.values(CAPS).sort());++});++++t('the service manifest proposal is a valid registry.service-manifest@1', () => {++ const m = JSON.parse(fs.readFileSync(path.join(DOCS, 'service-manifest-proposal.json'), 'utf8'));++ const v = contracts.validate('registry.service-manifest@1', m);++ assert.ok(v.valid, JSON.stringify(v.errors));++ assert.deepStrictEqual([...m.capabilities].sort(), Object.values(CAPS).sort());++});++++t('seeds: one real source per category, all disabled, all valid, notes recorded', () => {++ const seeds = require('../seeds/sources.json').sources;++ assert.deepStrictEqual(seeds.map(s => s.category).sort(), [...CATEGORIES].sort());++ for (const s of seeds) {++ const rec = validateSource(s);++ assert.strictEqual(rec.enabled, false, s.key);++ assert.ok(rec.terms_note, `${s.key} has a terms note`);++ if (rec.type !== 'manual') assert.ok(rec.robots_note, `${s.key} has a robots note`);++ assert.strictEqual(rec.default_indexability, 'noindex');++ for (const ep of rec.endpoints) assert.ok(ep.url.startsWith('https://'), ep.url);++ assert.ok(sourceSchema(rec), JSON.stringify(sourceSchema.errors));++ }++ const { openDb } = require('../server/db');++ const db = openDb(':memory:');++ assert.strictEqual(seed(db).created.length, seeds.length);++ assert.deepStrictEqual(seed(db).created, [], 'never overwrites');++ assert.strictEqual(db.prepare('SELECT COUNT(*) AS n FROM sources WHERE enabled = 1').get().n, 0);++ db.close();++});++++t('API views match sources.source@1 and sources.item@1', async () => {++ const svc = await boot();++ const web = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: rss([{ guid: 'p1', title: 'P', link: 'https://example.org/p' }]) }) });++ try {++ const admin = serviceToken('network', ['sources.*']);++ await request(svc.base, 'POST', '/api/v1/sources', { token: admin, body: sourceDef({ key: 'proposal-check', endpoints: [`${web.origin}/f`] }) });++ await request(svc.base, 'POST', '/api/v1/sources/proposal-check/fetch', { token: admin });++ const src = await request(svc.base, 'GET', '/api/v1/sources/proposal-check', { token: admin });++ assert.ok(sourceSchema(src.body.source), JSON.stringify(sourceSchema.errors));++ const items = await request(svc.base, 'GET', '/api/v1/items', { token: admin });++ for (const it of items.body.items) assert.ok(itemSchema(it), JSON.stringify(itemSchema.errors));++ const one = await request(svc.base, 'GET', `/api/v1/items/${items.body.items[0].id}?revisions=1`, { token: admin });++ assert.ok(itemSchema(one.body.item), JSON.stringify(itemSchema.errors));++ } finally { await svc.stop(); await web.close(); }++});++++t.run();nnew file mode 100644@@@ -0,0 +1,132 @@++'use strict';++/** robots.txt is respected (rules, unreachable = deny, redirects) and internal addresses are never fetched. */++const assert = require('assert');++const { parseRobots, isAllowed } = require('../server/robots');++const { isPublicAddress } = require('../server/net/guard');++const { boot, site, sourceDef, rss, suite } = require('./helpers');++++const t = suite('robots-ssrf');++const FEED = rss([{ guid: 'x', title: 'X', link: 'https://example.org/x' }]);++++t('robots rules: our group over *, longest match, Allow wins ties, * and $', () => {++ const txt = [++ 'User-agent: *', 'Disallow: /', '',++ 'User-agent: OpenVibeSources', 'User-agent: other-bot', 'Disallow: /private', 'Allow: /private/press', 'Disallow: /*.pdf$', 'Crawl-delay: 2', '',++ 'User-agent: openvibesources', 'Disallow: /tmp/',++ ].join('\n');++ const p = parseRobots(txt, 'OpenVibeSources');++ assert.strictEqual(p.crawlDelaySec, 2);++ assert.strictEqual(isAllowed(p, '/'), true, 'the * group does not apply when ours exists');++ assert.strictEqual(isAllowed(p, '/private/x'), false);++ assert.strictEqual(isAllowed(p, '/private/press/2026'), true);++ assert.strictEqual(isAllowed(p, '/docs/a.pdf'), false);++ assert.strictEqual(isAllowed(p, '/docs/a.pdf?x=1'), true);++ assert.strictEqual(isAllowed(p, '/tmp/file'), false, 'groups for the same agent merge');++ const star = parseRobots('User-agent: *\nDisallow: /a\nAllow: /a\n', 'OpenVibeSources');++ assert.strictEqual(isAllowed(star, '/a/b'), true, 'Allow wins a tie');++ const empty = parseRobots('User-agent: *\nDisallow:\n', 'OpenVibeSources');++ assert.strictEqual(isAllowed(empty, '/anything'), true);++ const prefix = parseRobots('User-agent: OpenVibe\nDisallow: /\n', 'OpenVibeSources');++ assert.strictEqual(isAllowed(prefix, '/x'), true, 'a different token (even a prefix of ours) does not apply');++});++++t('address policy', () => {++ for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '::1', 'fe80::1', 'fd00::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '64:ff9b::a00:1', '2002:c0a8:101::1', '224.0.0.1']) {++ assert.strictEqual(isPublicAddress(ip), false, ip);++ }++ for (const ip of ['93.184.216.34', '1.1.1.1', '2606:4700:4700::1111']) assert.strictEqual(isPublicAddress(ip), true, ip);++});++++let svc;++let web;++++t('boot', async () => {++ svc = await boot();++ web = await site({++ '/robots.txt': () => ({ body: 'User-agent: *\nDisallow: /private/\n' }),++ '/private/feed.xml': () => ({ body: FEED }),++ '/public/feed.xml': () => ({ body: FEED }),++ '/public/moved': () => ({ status: 301, headers: { Location: '/private/feed.xml' } }),++ });++});++++t('a disallowed endpoint is robots_denied and never requested', async () => {++ svc.registry.create(sourceDef({ key: 'denied', endpoints: [`${web.origin}/private/feed.xml`, `${web.origin}/public/feed.xml`] }), 'test');++ const out = await svc.ingest.run('denied', { trigger: 'manual' });++ assert.deepStrictEqual(out.runs.map(r => r.state), ['robots_denied', 'ok']);++ assert.strictEqual(web.hits('/private/feed.xml').length, 0);++ assert.strictEqual(web.hits('/robots.txt').length, 1, 'robots.txt is cached per origin');++ assert.ok(svc.outbox.all().some(e => e.event_type === 'sources.fetch.failed' && e.payload.state === 'robots_denied'));++});++++t('a redirect into a disallowed path is refused before it is followed', async () => {++ svc.registry.create(sourceDef({ key: 'hop', endpoints: [`${web.origin}/public/moved`] }), 'test');++ const out = await svc.ingest.run('hop', { trigger: 'manual' });++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['robots_denied', 'hop_refused']);++ assert.strictEqual(web.hits('/private/feed.xml').length, 0);++});++++t('robots.txt 5xx or unreachable means complete disallow; 404 means allowed', async () => {++ const down = await site({ '/robots.txt': () => ({ status: 503 }), '/feed.xml': () => ({ body: FEED }) });++ const none = await site({ '/robots.txt': () => ({ status: 404 }), '/feed.xml': () => ({ body: FEED }) });++ try {++ svc.registry.create(sourceDef({ key: 'robots-down', endpoints: [`${down.origin}/feed.xml`] }), 'test');++ svc.registry.create(sourceDef({ key: 'robots-none', endpoints: [`${none.origin}/feed.xml`] }), 'test');++ assert.strictEqual((await svc.ingest.run('robots-down', { trigger: 'manual' })).runs[0].state, 'robots_denied');++ assert.strictEqual(down.hits('/feed.xml').length, 0);++ assert.strictEqual((await svc.ingest.run('robots-none', { trigger: 'manual' })).runs[0].state, 'ok');++ } finally { await down.close(); await none.close(); }++});++++t('Crawl-delay spaces requests to the host', async () => {++ const s = await site({ '/robots.txt': () => ({ body: 'User-agent: *\nCrawl-delay: 1\n' }), '/a': () => ({ body: FEED }), '/b': () => ({ body: FEED }) });++ try {++ svc.registry.create(sourceDef({ key: 'crawl-delay', endpoints: [`${s.origin}/a`, `${s.origin}/b`] }), 'test');++ await svc.ingest.run('crawl-delay', { trigger: 'manual' });++ const gap = s.hits('/b')[0].at - s.hits('/a')[0].at;++ assert.ok(gap >= 950, `gap ${gap} ms`);++ } finally { await s.close(); }++});++++t('loopback and private addresses are refused unless explicitly allowlisted', async () => {++ const strict = await boot({ env: { SOURCES_ALLOW_PRIVATE_HOSTS: '' } });++ try {++ strict.registry.create(sourceDef({ key: 'loop', endpoints: [`${web.origin}/public/feed.xml`] }), 'test');++ const out = await strict.ingest.run('loop', { trigger: 'manual' });++ assert.strictEqual(out.runs[0].state, 'http_error');++ assert.ok(['address_refused', 'port_refused'].includes(out.runs[0].error_code), out.runs[0].error_code);++ const n = web.requests.length;++ strict.registry.create(sourceDef({ key: 'meta', endpoints: ['http://169.254.169.254/latest/meta-data/'] }), 'test');++ const meta = await strict.ingest.run('meta', { trigger: 'manual' });++ assert.deepStrictEqual([meta.runs[0].state, meta.runs[0].error_code], ['http_error', 'address_refused']);++ assert.strictEqual(web.requests.length, n);++ } finally { await strict.stop(); }++});++++t('a public-looking name that resolves to a private address is refused at connect time', async () => {++ // DNS stub: "feeds.example.com" resolves to 127.0.0.1 (a rebinding-style answer)++ const lookupImpl = (host, opts, cb) => cb(null, [{ address: '127.0.0.1', family: 4 }]);++ const strict = await boot({ env: { SOURCES_ALLOW_PRIVATE_HOSTS: '', SOURCES_ALLOWED_PORTS: `80,443,${new URL(web.origin).port}` }, lookupImpl });++ try {++ const port = new URL(web.origin).port;++ strict.registry.create(sourceDef({ key: 'rebind', endpoints: [`http://feeds.example.com:${port}/public/feed.xml`] }), 'test');++ const n = web.requests.length;++ const out = await strict.ingest.run('rebind', { trigger: 'manual' });++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['http_error', 'address_refused']);++ assert.strictEqual(web.requests.length, n, 'no connection was made');++ assert.strictEqual(strict.db.prepare('SELECT COUNT(*) AS n FROM items').get().n, 0);++ } finally { await strict.stop(); }++});++++t('redirects to an internal address are refused', async () => {++ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/r': () => ({ status: 302, headers: { Location: 'http://10.0.0.5/admin' } }) });++ try {++ svc.registry.create(sourceDef({ key: 'redir-internal', endpoints: [`${s.origin}/r`] }), 'test');++ const out = await svc.ingest.run('redir-internal', { trigger: 'manual' });++ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['http_error', 'address_refused']);++ } finally { await s.close(); }++});++++t('done', async () => { await web.close(); await svc.stop(); });++++t.run();nnew file mode 100644@@@ -0,0 +1,56 @@++#!/usr/bin/env node++/**++ * Runs every test in test/ — the files named *.test.js — each in its own process, and fails++ * if any of them fails. They use temp SQLite databases, generated RSA keys and stub subscribers++ * on random ports; none of them needs the network or a running OpenVibe.Network.++ *++ * npm test # everything++ * npm test -- publish sse # only files whose name contains one of the words++ */++'use strict';++const { spawn } = require('child_process');++const fs = require('fs');++const path = require('path');++++const filters = process.argv.slice(2).filter((a) => !a.startsWith('-'));++const files = fs.readdirSync(__dirname)++ .filter((f) => f.endsWith('.test.js'))++ .filter((f) => !filters.length || filters.some((w) => f.includes(w)))++ .sort();++const TIMEOUT_MS = 60000;++++function runOne(file) {++ return new Promise((resolve) => {++ const started = Date.now();++ const child = spawn(process.execPath, [path.join(__dirname, file)], {++ cwd: path.join(__dirname, '..'),++ env: { ...process.env, NODE_ENV: 'test' },++ stdio: ['ignore', 'pipe', 'pipe'],++ });++ let output = '';++ child.stdout.on('data', (c) => { output += c; });++ child.stderr.on('data', (c) => { output += c; });++ const timer = setTimeout(() => { output += `\n[run] timed out after ${TIMEOUT_MS}ms`; child.kill('SIGKILL'); }, TIMEOUT_MS);++ child.on('close', (code, signal) => {++ clearTimeout(timer);++ resolve({ file, ok: code === 0, code: code ?? signal, ms: Date.now() - started, output });++ });++ });++}++++(async () => {++ if (!files.length) { console.error('no test files matched'); process.exit(1); }++ const results = [];++ for (const f of files) {++ const r = await runOne(f);++ results.push(r);++ console.log(`${r.ok ? '✓' : '✗'} ${r.file.padEnd(32)} ${String(r.ms).padStart(6)}ms`);++ }++ const failed = results.filter((r) => !r.ok);++ for (const r of failed) {++ console.log(`\n── ${r.file} (exit ${r.code}) ──`);++ console.log(r.output.split('\n').slice(-60).join('\n'));++ }++ console.log(`\n${results.length - failed.length}/${results.length} test files passed`);++ process.exit(failed.length ? 1 : 0);++})();nnew file mode 100644@@@ -0,0 +1,85 @@++'use strict';++/** The in-process scheduler, failure backoff, and the outbox relay to OpenVibe.Events. */++const assert = require('assert');++const http = require('http');++const { validate } = require('openvibe-contracts');++const { boot, site, sourceDef, rss, sleep, suite } = require('./helpers');++++const t = suite('scheduler-events');++const FEED = rss([{ guid: 's1', title: 'Scheduled', link: 'https://example.org/s1' }]);++++async function waitFor(fn, ms = 3000) {++ const end = Date.now() + ms;++ while (Date.now() < end) { const v = fn(); if (v) return v; await sleep(25); }++ throw new Error('timed out waiting');++}++++t('due, enabled sources are fetched on schedule; disabled and manual ones never are', async () => {++ let status = 200;++ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => (status === 200 ? { body: FEED } : { status }) });++ const svc = await boot({ worker: 'on' });++ try {++ svc.registry.create(sourceDef({ key: 'sched', endpoints: [`${s.origin}/f`], poll_interval_sec: 3600 }), 'test');++ svc.registry.create(sourceDef({ key: 'off', endpoints: [`${s.origin}/f`], enabled: false }), 'test');++ svc.registry.create({ key: 'hand', name: 'Manual', type: 'manual', category: 'coupons', terms_note: 'x', enabled: true }, 'test');++ await waitFor(() => svc.db.prepare("SELECT 1 FROM fetch_runs WHERE source_key = 'sched'").get());++ const row = svc.registry.get('sched');++ assert.strictEqual(row.last_state, 'ok');++ assert.ok(row.next_due_at - Date.now() > 3500 * 1000, 'next poll one interval later');++ await sleep(200);++ assert.strictEqual(svc.db.prepare("SELECT COUNT(*) AS n FROM fetch_runs WHERE source_key IN ('off', 'hand')").get().n, 0);++ assert.strictEqual(svc.db.prepare("SELECT COUNT(*) AS n FROM fetch_runs WHERE source_key = 'sched'").get().n, 1, 'not refetched before it is due');++++ // a failing source backs off exponentially++ status = 500;++ svc.db.prepare("UPDATE sources SET next_due_at = 0, last_request_at = NULL WHERE key = 'sched'").run();++ await waitFor(() => svc.registry.get('sched').consecutive_failures === 1);++ const f1 = svc.registry.get('sched').next_due_at - svc.registry.get('sched').last_run_at;++ svc.db.prepare("UPDATE sources SET next_due_at = 0, last_request_at = NULL WHERE key = 'sched'").run();++ await waitFor(() => svc.registry.get('sched').consecutive_failures === 2);++ const f2 = svc.registry.get('sched').next_due_at - svc.registry.get('sched').last_run_at;++ assert.strictEqual(f1, 3600 * 1000);++ assert.strictEqual(f2, 2 * 3600 * 1000);++++ // not_before (Retry-After) holds the scheduler back++ svc.db.prepare("UPDATE sources SET next_due_at = 0, not_before = ?, last_request_at = NULL WHERE key = 'sched'").run(Date.now() + 60000);++ const n = s.hits('/f').length;++ await sleep(300);++ assert.strictEqual(s.hits('/f').length, n);++ } finally { await svc.stop(); await s.close(); }++});++++t('events are valid envelopes and reach OpenVibe.Events through the relay', async () => {++ const received = [];++ const events = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', c => chunks.push(c));++ req.on('end', () => {++ const body = JSON.parse(Buffer.concat(chunks).toString('utf8'));++ received.push(...(body.events || [body]));++ res.statusCode = 201;++ res.end('{}');++ });++ });++ await new Promise(r => events.listen(0, '127.0.0.1', r));++ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: FEED }), '/bad': () => ({ status: 502 }) });++ const tokenClient = { async authHeaders() { return { Authorization: 'Bearer relay' }; }, invalidate() {} };++ const svc = await boot({ env: { EVENTS_URL: `http://127.0.0.1:${events.address().port}` }, tokenClient });++ try {++ svc.registry.create(sourceDef({ key: 'ev', endpoints: [`${s.origin}/f`, `${s.origin}/bad`] }), 'test');++ await svc.ingest.run('ev', { trigger: 'manual' });++ for (let i = 0; i < 5 && svc.outbox.pending(); i++) await svc.relay.flush();++ assert.strictEqual(svc.outbox.pending(), 0);++ assert.deepStrictEqual(received.map(e => e.event_type).sort(), ['sources.fetch.failed', 'sources.item.created']);++ for (const e of received) {++ const v = validate('events.event-envelope@1', e);++ assert.ok(v.valid, JSON.stringify(v.errors));++ assert.strictEqual(e.source, 'sources');++ assert.strictEqual(e.visibility, 'internal');++ }++ const failed = received.find(e => e.event_type === 'sources.fetch.failed');++ assert.deepStrictEqual([failed.payload.state, failed.payload.http_status, failed.payload.error_code], ['http_error', 502, 'http_502']);++ } finally { await svc.stop(); await s.close(); await new Promise(r => events.close(() => r())); }++});++++t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,12 @@
@@ -0,0 +1,12 @@
+node_modules/
+data/
+dist/
+.env
+.env.*
+!.env.example
+*.db
+*.db-wal
+*.db-shm
+*.log
+*.pem
+*.key
new file mode 100644
new file mode 100644
@@ -0,0 +1,661 @@
@@ -0,0 +1,661 @@
+ GNU AFFERO GENERAL PUBLIC LICENSE
+ Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+ The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+ Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+ A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate. Many developers of free software are heartened and
+encouraged by the resulting cooperation. However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+ The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community. It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server. Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+ An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals. This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ TERMS AND CONDITIONS
+
+ 0. Definitions.
+
+ "This License" refers to version 3 of the GNU Affero General Public License.
+
+ "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+ "The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+ To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy. The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+ A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+ To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+ To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+ An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+ 1. Source Code.
+
+ The "source code" for a work means the preferred form of the work
+for making modifications to it. "Object code" means any non-source
+form of a work.
+
+ A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+ The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+ The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+ The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+ The Corresponding Source for a work in source code form is that
+same work.
+
+ 2. Basic Permissions.
+
+ All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+ You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force. You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright. Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+ Conveying under any other circumstances is permitted solely under
+the conditions stated below. Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+ No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+ When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+ 4. Conveying Verbatim Copies.
+
+ You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+ You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+ 5. Conveying Modified Source Versions.
+
+ You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+ a) The work must carry prominent notices stating that you modified
+ it, and giving a relevant date.
+
+ b) The work must carry prominent notices stating that it is
+ released under this License and any conditions added under section
+ 7. This requirement modifies the requirement in section 4 to
+ "keep intact all notices".
+
+ c) You must license the entire work, as a whole, under this
+ License to anyone who comes into possession of a copy. This
+ License will therefore apply, along with any applicable section 7
+ additional terms, to the whole of the work, and all its parts,
+ regardless of how they are packaged. This License gives no
+ permission to license the work in any other way, but it does not
+ invalidate such permission if you have separately received it.
+
+ d) If the work has interactive user interfaces, each must display
+ Appropriate Legal Notices; however, if the Program has interactive
+ interfaces that do not display Appropriate Legal Notices, your
+ work need not make them do so.
+
+ A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+ 6. Conveying Non-Source Forms.
+
+ You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+ a) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by the
+ Corresponding Source fixed on a durable physical medium
+ customarily used for software interchange.
+
+ b) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by a
+ written offer, valid for at least three years and valid for as
+ long as you offer spare parts or customer support for that product
+ model, to give anyone who possesses the object code either (1) a
+ copy of the Corresponding Source for all the software in the
+ product that is covered by this License, on a durable physical
+ medium customarily used for software interchange, for a price no
+ more than your reasonable cost of physically performing this
+ conveying of source, or (2) access to copy the
+ Corresponding Source from a network server at no charge.
+
+ c) Convey individual copies of the object code with a copy of the
+ written offer to provide the Corresponding Source. This
+ alternative is allowed only occasionally and noncommercially, and
+ only if you received the object code with such an offer, in accord
+ with subsection 6b.
+
+ d) Convey the object code by offering access from a designated
+ place (gratis or for a charge), and offer equivalent access to the
+ Corresponding Source in the same way through the same place at no
+ further charge. You need not require recipients to copy the
+ Corresponding Source along with the object code. If the place to
+ copy the object code is a network server, the Corresponding Source
+ may be on a different server (operated by you or a third party)
+ that supports equivalent copying facilities, provided you maintain
+ clear directions next to the object code saying where to find the
+ Corresponding Source. Regardless of what server hosts the
+ Corresponding Source, you remain obligated to ensure that it is
+ available for as long as needed to satisfy these requirements.
+
+ e) Convey the object code using peer-to-peer transmission, provided
+ you inform other peers where the object code and Corresponding
+ Source of the work are being offered to the general public at no
+ charge under subsection 6d.
+
+ A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+ A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling. In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage. For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product. A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+ "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source. The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+ If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+ The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed. Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+ Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+ 7. Additional Terms.
+
+ "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+ When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+ Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+ a) Disclaiming warranty or limiting liability differently from the
+ terms of sections 15 and 16 of this License; or
+
+ b) Requiring preservation of specified reasonable legal notices or
+ author attributions in that material or in the Appropriate Legal
+ Notices displayed by works containing it; or
+
+ c) Prohibiting misrepresentation of the origin of that material, or
+ requiring that modified versions of such material be marked in
+ reasonable ways as different from the original version; or
+
+ d) Limiting the use for publicity purposes of names of licensors or
+ authors of the material; or
+
+ e) Declining to grant rights under trademark law for use of some
+ trade names, trademarks, or service marks; or
+
+ f) Requiring indemnification of licensors and authors of that
+ material by anyone who conveys the material (or modified versions of
+ it) with contractual assumptions of liability to the recipient, for
+ any liability that these contractual assumptions directly impose on
+ those licensors and authors.
+
+ All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+ If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+ Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+ 8. Termination.
+
+ You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+ However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+ Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+ Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+ 9. Acceptance Not Required for Having Copies.
+
+ You are not required to accept this License in order to receive or
+run a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+ 10. Automatic Licensing of Downstream Recipients.
+
+ Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+ An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+ You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+ 11. Patents.
+
+ A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+ A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+ In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+ If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+ If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+ A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License. You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+ Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+ 12. No Surrender of Others' Freedom.
+
+ If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all. For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+ 13. Remote Network Interaction; Use with the GNU General Public License.
+
+ Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software. This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+ Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+ 14. Revised Versions of this License.
+
+ The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time. Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+ If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+ Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+ 15. Disclaimer of Warranty.
+
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. Limitation of Liability.
+
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+ 17. Interpretation of Sections 15 and 16.
+
+ If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+ <one line to give the program's name and a brief idea of what it does.>
+ Copyright (C) <year> <name of author>
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU Affero General Public License as published
+ by the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU Affero General Public License for more details.
+
+ You should have received a copy of the GNU Affero General Public License
+ along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+ If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source. For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code. There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+ You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<https://www.gnu.org/licenses/>.
new file mode 100644
new file mode 100644
@@ -0,0 +1,1395 @@
@@ -0,0 +1,1395 @@
+{
+ "name": "openvibe-sources",
+ "version": "0.1.0",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "openvibe-sources",
+ "version": "0.1.0",
+ "license": "AGPL-3.0-only",
+ "dependencies": {
+ "better-sqlite3": "^11.10.0",
+ "dotenv": "^16.4.7",
+ "express": "^4.21.2",
+ "fast-xml-parser": "^4.5.3",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0"
+ },
+ "devDependencies": {
+ "ajv": "^8.20.0",
+ "ajv-formats": "^3.0.1"
+ },
+ "engines": {
+ "node": ">=22.0.0"
+ }
+ },
+ "node_modules/accepts": {
+ "version": "1.3.8",
+ "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
+ "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==",
+ "license": "MIT",
+ "dependencies": {
+ "mime-types": "~2.1.34",
+ "negotiator": "0.6.3"
+ },
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/ajv": {
+ "version": "8.20.0",
+ "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz",
+ "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==",
+ "license": "MIT",
+ "dependencies": {
+ "fast-deep-equal": "^3.1.3",
+ "fast-uri": "^3.0.1",
+ "json-schema-traverse": "^1.0.0",
+ "require-from-string": "^2.0.2"
+ },
+ "funding": {
+ "type": "github",
+ "url": "https://github.com/sponsors/epoberezkin"
+ }
+ },
+ "node_modules/ajv-formats": {
+ "version": "3.0.1",
+ "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz",
+ "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==",
+ "license": "MIT",
+ "dependencies": {
+ "ajv": "^8.0.0"
+ },
+ "peerDependencies": {
+ "ajv": "^8.0.0"
+ },
+ "peerDependenciesMeta": {
+ "ajv": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/array-flatten": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz",
+ "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",
+ "license": "MIT"
+ },
+ "node_modules/base64-js": {
+ "version": "1.5.1",
+ "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
+ "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "MIT"
+ },
+ "node_modules/better-sqlite3": {
+ "version": "11.10.0",
+ "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz",
+ "integrity": "sha512-EwhOpyXiOEL/lKzHz9AW1msWFNzGc/z+LzeB3/jnFJpxu+th2yqvzsSWas1v9jgs9+xiXJcD5A8CJxAG2TaghQ==",
+ "hasInstallScript": true,
+ "license": "MIT",
+ "dependencies": {
+ "bindings": "^1.5.0",
+ "prebuild-install": "^7.1.1"
+ }
+ },
+ "node_modules/bindings": {
+ "version": "1.5.0",
+ "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz",
+ "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==",
+ "license": "MIT",
+ "dependencies": {
+ "file-uri-to-path": "1.0.0"
+ }
+ },
+ "node_modules/bl": {
+ "version": "4.1.0",
+ "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
+ "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",
+ "license": "MIT",
+ "dependencies": {
+ "buffer": "^5.5.0",
+ "inherits": "^2.0.4",
+ "readable-stream": "^3.4.0"
+ }
+ },
+ "node_modules/body-parser": {
+ "version": "1.20.8",
+ "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
+ "integrity": "sha512-JNcyFQ64OiijEkPzUBTCe+hyPXUD/3LEldGQ6iF5LR1w00mx9o7xtDWHXBY2iItjdCFGoilOLNQbH943ut7pHA==",
+ "license": "MIT",
+ "dependencies": {
+ "bytes": "~3.1.2",
+ "content-type": "~1.0.5",
+ "debug": "2.6.9",
+ "depd": "2.0.0",
+ "destroy": "~1.2.0",
+ "http-errors": "~2.0.1",
+ "iconv-lite": "~0.4.24",
+ "on-finished": "~2.4.1",
+ "qs": "~6.16.0",
+ "raw-body": "~2.5.3",
+ "type-is": "~1.6.18",
+ "unpipe": "~1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.8",
+ "npm": "1.2.8000 || >= 1.4.16"
+ }
+ },
+ "node_modules/buffer": {
+ "version": "5.7.1",
+ "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",
+ "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "MIT",
+ "dependencies": {
+ "base64-js": "^1.3.1",
+ "ieee754": "^1.1.13"
+ }
+ },
+ "node_modules/bytes": {
+ "version": "3.1.2",
+ "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
+ "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/call-bind-apply-helpers": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
+ "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
+ "license": "MIT",
+ "dependencies": {
+ "es-errors": "^1.3.0",
+ "function-bind": "^1.1.2"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/call-bound": {
+ "version": "1.0.4",
+ "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",
+ "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",
+ "license": "MIT",
+ "dependencies": {
+ "call-bind-apply-helpers": "^1.0.2",
+ "get-intrinsic": "^1.3.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/chownr": {
+ "version": "1.1.4",
+ "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",
+ "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",
+ "license": "ISC"
+ },
+ "node_modules/content-disposition": {
+ "version": "0.5.4",
+ "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz",
+ "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==",
+ "license": "MIT",
+ "dependencies": {
+ "safe-buffer": "5.2.1"
+ },
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/content-type": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
+ "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/cookie": {
+ "version": "0.7.2",
+ "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
+ "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/cookie-signature": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.7.tgz",
+ "integrity": "sha512-NXdYc3dLr47pBkpUCHtKSwIOQXLVn8dZEuywboCOJY/osA0wFSLlSawr3KN8qXJEyX66FcONTH8EIlVuK0yyFA==",
+ "license": "MIT"
+ },
+ "node_modules/debug": {
+ "version": "2.6.9",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+ "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+ "license": "MIT",
+ "dependencies": {
+ "ms": "2.0.0"
+ }
+ },
+ "node_modules/decompress-response": {
+ "version": "6.0.0",
+ "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",
+ "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",
+ "license": "MIT",
+ "dependencies": {
+ "mimic-response": "^3.1.0"
+ },
+ "engines": {
+ "node": ">=10"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/sindresorhus"
+ }
+ },
+ "node_modules/deep-extend": {
+ "version": "0.6.0",
+ "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",
+ "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/depd": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
+ "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/destroy": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz",
+ "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8",
+ "npm": "1.2.8000 || >= 1.4.16"
+ }
+ },
+ "node_modules/detect-libc": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=8"
+ }
+ },
+ "node_modules/dotenv": {
+ "version": "16.6.1",
+ "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",
+ "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==",
+ "license": "BSD-2-Clause",
+ "engines": {
+ "node": ">=12"
+ },
+ "funding": {
+ "url": "https://dotenvx.com"
+ }
+ },
+ "node_modules/dunder-proto": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
+ "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
+ "license": "MIT",
+ "dependencies": {
+ "call-bind-apply-helpers": "^1.0.1",
+ "es-errors": "^1.3.0",
+ "gopd": "^1.2.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/ee-first": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
+ "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==",
+ "license": "MIT"
+ },
+ "node_modules/encodeurl": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
+ "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/end-of-stream": {
+ "version": "1.4.5",
+ "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
+ "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
+ "license": "MIT",
+ "dependencies": {
+ "once": "^1.4.0"
+ }
+ },
+ "node_modules/es-define-property": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
+ "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/es-errors": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
+ "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/es-object-atoms": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz",
+ "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==",
+ "license": "MIT",
+ "dependencies": {
+ "es-errors": "^1.3.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/escape-html": {
+ "version": "1.0.3",
+ "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
+ "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==",
+ "license": "MIT"
+ },
+ "node_modules/etag": {
+ "version": "1.8.1",
+ "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
+ "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/expand-template": {
+ "version": "2.0.3",
+ "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",
+ "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",
+ "license": "(MIT OR WTFPL)",
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/express": {
+ "version": "4.22.3",
+ "resolved": "https://registry.npmjs.org/express/-/express-4.22.3.tgz",
+ "integrity": "sha512-Bdcs4+3qlpVlx2NRn6fgX2Ue2/gGRaPeawebgclM0ERSCqDpA+owF1fdPwjJUTAJWMTuAaxjDf+hzb0/4eKvvw==",
+ "license": "MIT",
+ "dependencies": {
+ "accepts": "~1.3.8",
+ "array-flatten": "1.1.1",
+ "body-parser": "~1.20.5",
+ "content-disposition": "~0.5.4",
+ "content-type": "~1.0.4",
+ "cookie": "~0.7.1",
+ "cookie-signature": "~1.0.6",
+ "debug": "2.6.9",
+ "depd": "2.0.0",
+ "encodeurl": "~2.0.0",
+ "escape-html": "~1.0.3",
+ "etag": "~1.8.1",
+ "finalhandler": "~1.3.1",
+ "fresh": "~0.5.2",
+ "http-errors": "~2.0.0",
+ "merge-descriptors": "1.0.3",
+ "methods": "~1.1.2",
+ "on-finished": "~2.4.1",
+ "parseurl": "~1.3.3",
+ "path-to-regexp": "~0.1.13",
+ "proxy-addr": "~2.0.7",
+ "qs": "~6.16.0",
+ "range-parser": "~1.2.1",
+ "safe-buffer": "5.2.1",
+ "send": "~0.19.0",
+ "serve-static": "~1.16.2",
+ "setprototypeof": "1.2.0",
+ "statuses": "~2.0.1",
+ "type-is": "~1.6.18",
+ "utils-merge": "1.0.1",
+ "vary": "~1.1.2"
+ },
+ "engines": {
+ "node": ">= 0.10.0"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/express"
+ }
+ },
+ "node_modules/fast-deep-equal": {
+ "version": "3.1.3",
+ "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",
+ "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==",
+ "license": "MIT"
+ },
+ "node_modules/fast-uri": {
+ "version": "3.1.8",
+ "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz",
+ "integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/fastify"
+ },
+ {
+ "type": "opencollective",
+ "url": "https://opencollective.com/fastify"
+ }
+ ],
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/fast-xml-parser": {
+ "version": "4.5.7",
+ "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-4.5.7.tgz",
+ "integrity": "sha512-a6Qh1RMCNbSrU1+sAyAAZH3rTe+OaWJbNZIq0S+ifZciUUOQtlVxBJwoTUE2bYhysmG/RYyI5WJFIKdBahJdrQ==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/NaturalIntelligence"
+ }
+ ],
+ "license": "MIT",
+ "dependencies": {
+ "strnum": "^1.0.5"
+ },
+ "bin": {
+ "fxparser": "src/cli/cli.js"
+ }
+ },
+ "node_modules/file-uri-to-path": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz",
+ "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==",
+ "license": "MIT"
+ },
+ "node_modules/finalhandler": {
+ "version": "1.3.2",
+ "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",
+ "integrity": "sha512-aA4RyPcd3badbdABGDuTXCMTtOneUCAYH/gxoYRTZlIJdF0YPWuGqiAsIrhNnnqdXGswYk6dGujem4w80UJFhg==",
+ "license": "MIT",
+ "dependencies": {
+ "debug": "2.6.9",
+ "encodeurl": "~2.0.0",
+ "escape-html": "~1.0.3",
+ "on-finished": "~2.4.1",
+ "parseurl": "~1.3.3",
+ "statuses": "~2.0.2",
+ "unpipe": "~1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/forwarded": {
+ "version": "0.2.0",
+ "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
+ "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/fresh": {
+ "version": "0.5.2",
+ "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz",
+ "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/fs-constants": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",
+ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",
+ "license": "MIT"
+ },
+ "node_modules/function-bind": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
+ "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
+ "license": "MIT",
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/get-intrinsic": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
+ "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
+ "license": "MIT",
+ "dependencies": {
+ "call-bind-apply-helpers": "^1.0.2",
+ "es-define-property": "^1.0.1",
+ "es-errors": "^1.3.0",
+ "es-object-atoms": "^1.1.1",
+ "function-bind": "^1.1.2",
+ "get-proto": "^1.0.1",
+ "gopd": "^1.2.0",
+ "has-symbols": "^1.1.0",
+ "hasown": "^2.0.2",
+ "math-intrinsics": "^1.1.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/get-proto": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
+ "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
+ "license": "MIT",
+ "dependencies": {
+ "dunder-proto": "^1.0.1",
+ "es-object-atoms": "^1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/github-from-package": {
+ "version": "0.0.0",
+ "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",
+ "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",
+ "license": "MIT"
+ },
+ "node_modules/gopd": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
+ "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/has-symbols": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
+ "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/hasown": {
+ "version": "2.0.4",
+ "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
+ "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
+ "license": "MIT",
+ "dependencies": {
+ "function-bind": "^1.1.2"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/http-errors": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
+ "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==",
+ "license": "MIT",
+ "dependencies": {
+ "depd": "~2.0.0",
+ "inherits": "~2.0.4",
+ "setprototypeof": "~1.2.0",
+ "statuses": "~2.0.2",
+ "toidentifier": "~1.0.1"
+ },
+ "engines": {
+ "node": ">= 0.8"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/express"
+ }
+ },
+ "node_modules/iconv-lite": {
+ "version": "0.4.24",
+ "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
+ "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
+ "license": "MIT",
+ "dependencies": {
+ "safer-buffer": ">= 2.1.2 < 3"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/ieee754": {
+ "version": "1.2.1",
+ "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
+ "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "BSD-3-Clause"
+ },
+ "node_modules/inherits": {
+ "version": "2.0.4",
+ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
+ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
+ "license": "ISC"
+ },
+ "node_modules/ini": {
+ "version": "1.3.8",
+ "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
+ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
+ "license": "ISC"
+ },
+ "node_modules/ipaddr.js": {
+ "version": "1.9.1",
+ "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
+ "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.10"
+ }
+ },
+ "node_modules/json-schema-traverse": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz",
+ "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==",
+ "license": "MIT"
+ },
+ "node_modules/math-intrinsics": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
+ "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/media-typer": {
+ "version": "0.3.0",
+ "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
+ "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/merge-descriptors": {
+ "version": "1.0.3",
+ "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",
+ "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",
+ "license": "MIT",
+ "funding": {
+ "url": "https://github.com/sponsors/sindresorhus"
+ }
+ },
+ "node_modules/methods": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz",
+ "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/mime": {
+ "version": "1.6.0",
+ "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
+ "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
+ "license": "MIT",
+ "bin": {
+ "mime": "cli.js"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/mime-db": {
+ "version": "1.52.0",
+ "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
+ "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/mime-types": {
+ "version": "2.1.35",
+ "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
+ "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
+ "license": "MIT",
+ "dependencies": {
+ "mime-db": "1.52.0"
+ },
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/mimic-response": {
+ "version": "3.1.0",
+ "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",
+ "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=10"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/sindresorhus"
+ }
+ },
+ "node_modules/minimist": {
+ "version": "1.2.8",
+ "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
+ "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
+ "license": "MIT",
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/mkdirp-classic": {
+ "version": "0.5.3",
+ "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
+ "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",
+ "license": "MIT"
+ },
+ "node_modules/ms": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+ "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
+ "license": "MIT"
+ },
+ "node_modules/napi-build-utils": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",
+ "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",
+ "license": "MIT"
+ },
+ "node_modules/negotiator": {
+ "version": "0.6.3",
+ "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz",
+ "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/node-abi": {
+ "version": "3.96.0",
+ "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.96.0.tgz",
+ "integrity": "sha512-rebQ/lz7i0EkoLzUVSrKRzA69zMkwLp95kKMWoMDkkM00Suxz0D7zEQPwRml5fQum24mj7bPvmlgLAmu2JCiYg==",
+ "license": "MIT",
+ "dependencies": {
+ "semver": "^7.3.5"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/object-inspect": {
+ "version": "1.13.4",
+ "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
+ "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/on-finished": {
+ "version": "2.4.1",
+ "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
+ "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
+ "license": "MIT",
+ "dependencies": {
+ "ee-first": "1.1.1"
+ },
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/once": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
+ "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
+ "license": "ISC",
+ "dependencies": {
+ "wrappy": "1"
+ }
+ },
+ "node_modules/openvibe-contracts": {
+ "version": "0.7.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0",
+ "integrity": "sha512-gUrjxbTy/HX0QMQGoDPphmFdwm7iitmNMMchs9tDSDGR47FRFHL/RgBgIKYR1S7M6ICu261YycFV8+YVglafXA==",
+ "license": "AGPL-3.0-only",
+ "dependencies": {
+ "ajv": "^8.17.1",
+ "ajv-formats": "^3.0.1"
+ },
+ "bin": {
+ "openvibe-contracts-check": "bin/check-service.js"
+ },
+ "engines": {
+ "node": ">=20"
+ }
+ },
+ "node_modules/parseurl": {
+ "version": "1.3.3",
+ "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
+ "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/path-to-regexp": {
+ "version": "0.1.13",
+ "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.13.tgz",
+ "integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
+ "license": "MIT"
+ },
+ "node_modules/prebuild-install": {
+ "version": "7.1.3",
+ "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
+ "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",
+ "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",
+ "license": "MIT",
+ "dependencies": {
+ "detect-libc": "^2.0.0",
+ "expand-template": "^2.0.3",
+ "github-from-package": "0.0.0",
+ "minimist": "^1.2.3",
+ "mkdirp-classic": "^0.5.3",
+ "napi-build-utils": "^2.0.0",
+ "node-abi": "^3.3.0",
+ "pump": "^3.0.0",
+ "rc": "^1.2.7",
+ "simple-get": "^4.0.0",
+ "tar-fs": "^2.0.0",
+ "tunnel-agent": "^0.6.0"
+ },
+ "bin": {
+ "prebuild-install": "bin.js"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/proxy-addr": {
+ "version": "2.0.8",
+ "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",
+ "integrity": "sha512-5nnx0yGyVUcY6t9RnWcARWtwT9F1D8O9rt08htPvnd49W1IgZtmLkhu9WfMzQj1cFxjHIO6connUNVW5k7AVyQ==",
+ "license": "MIT",
+ "dependencies": {
+ "forwarded": "0.2.0",
+ "ipaddr.js": "1.9.1"
+ },
+ "engines": {
+ "node": ">= 0.10"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/express"
+ }
+ },
+ "node_modules/pump": {
+ "version": "3.0.4",
+ "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
+ "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
+ "license": "MIT",
+ "dependencies": {
+ "end-of-stream": "^1.1.0",
+ "once": "^1.3.1"
+ }
+ },
+ "node_modules/qs": {
+ "version": "6.16.0",
+ "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
+ "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",
+ "license": "BSD-3-Clause",
+ "dependencies": {
+ "es-define-property": "^1.0.1",
+ "side-channel": "^1.1.1"
+ },
+ "engines": {
+ "node": ">=0.6"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/range-parser": {
+ "version": "1.2.1",
+ "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
+ "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/raw-body": {
+ "version": "2.5.3",
+ "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz",
+ "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==",
+ "license": "MIT",
+ "dependencies": {
+ "bytes": "~3.1.2",
+ "http-errors": "~2.0.1",
+ "iconv-lite": "~0.4.24",
+ "unpipe": "~1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/rc": {
+ "version": "1.2.8",
+ "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",
+ "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",
+ "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",
+ "dependencies": {
+ "deep-extend": "^0.6.0",
+ "ini": "~1.3.0",
+ "minimist": "^1.2.0",
+ "strip-json-comments": "~2.0.1"
+ },
+ "bin": {
+ "rc": "cli.js"
+ }
+ },
+ "node_modules/readable-stream": {
+ "version": "3.6.2",
+ "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
+ "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
+ "license": "MIT",
+ "dependencies": {
+ "inherits": "^2.0.3",
+ "string_decoder": "^1.1.1",
+ "util-deprecate": "^1.0.1"
+ },
+ "engines": {
+ "node": ">= 6"
+ }
+ },
+ "node_modules/require-from-string": {
+ "version": "2.0.2",
+ "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
+ "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/safe-buffer": {
+ "version": "5.2.1",
+ "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
+ "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "MIT"
+ },
+ "node_modules/safer-buffer": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+ "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
+ "license": "MIT"
+ },
+ "node_modules/semver": {
+ "version": "7.8.5",
+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz",
+ "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==",
+ "license": "ISC",
+ "bin": {
+ "semver": "bin/semver.js"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
+ "node_modules/send": {
+ "version": "0.19.2",
+ "resolved": "https://registry.npmjs.org/send/-/send-0.19.2.tgz",
+ "integrity": "sha512-VMbMxbDeehAxpOtWJXlcUS5E8iXh6QmN+BkRX1GARS3wRaXEEgzCcB10gTQazO42tpNIya8xIyNx8fll1OFPrg==",
+ "license": "MIT",
+ "dependencies": {
+ "debug": "2.6.9",
+ "depd": "2.0.0",
+ "destroy": "1.2.0",
+ "encodeurl": "~2.0.0",
+ "escape-html": "~1.0.3",
+ "etag": "~1.8.1",
+ "fresh": "~0.5.2",
+ "http-errors": "~2.0.1",
+ "mime": "1.6.0",
+ "ms": "2.1.3",
+ "on-finished": "~2.4.1",
+ "range-parser": "~1.2.1",
+ "statuses": "~2.0.2"
+ },
+ "engines": {
+ "node": ">= 0.8.0"
+ }
+ },
+ "node_modules/send/node_modules/ms": {
+ "version": "2.1.3",
+ "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
+ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
+ "license": "MIT"
+ },
+ "node_modules/serve-static": {
+ "version": "1.16.3",
+ "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz",
+ "integrity": "sha512-x0RTqQel6g5SY7Lg6ZreMmsOzncHFU7nhnRWkKgWuMTu5NN0DR5oruckMqRvacAN9d5w6ARnRBXl9xhDCgfMeA==",
+ "license": "MIT",
+ "dependencies": {
+ "encodeurl": "~2.0.0",
+ "escape-html": "~1.0.3",
+ "parseurl": "~1.3.3",
+ "send": "~0.19.1"
+ },
+ "engines": {
+ "node": ">= 0.8.0"
+ }
+ },
+ "node_modules/setprototypeof": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
+ "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
+ "license": "ISC"
+ },
+ "node_modules/side-channel": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
+ "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
+ "license": "MIT",
+ "dependencies": {
+ "es-errors": "^1.3.0",
+ "object-inspect": "^1.13.4",
+ "side-channel-list": "^1.0.1",
+ "side-channel-map": "^1.0.1",
+ "side-channel-weakmap": "^1.0.2"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/side-channel-list": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
+ "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
+ "license": "MIT",
+ "dependencies": {
+ "es-errors": "^1.3.0",
+ "object-inspect": "^1.13.4"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/side-channel-map": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
+ "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
+ "license": "MIT",
+ "dependencies": {
+ "call-bound": "^1.0.2",
+ "es-errors": "^1.3.0",
+ "get-intrinsic": "^1.2.5",
+ "object-inspect": "^1.13.3"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/side-channel-weakmap": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
+ "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
+ "license": "MIT",
+ "dependencies": {
+ "call-bound": "^1.0.2",
+ "es-errors": "^1.3.0",
+ "get-intrinsic": "^1.2.5",
+ "object-inspect": "^1.13.3",
+ "side-channel-map": "^1.0.1"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/simple-concat": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",
+ "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "MIT"
+ },
+ "node_modules/simple-get": {
+ "version": "4.0.1",
+ "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",
+ "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "license": "MIT",
+ "dependencies": {
+ "decompress-response": "^6.0.0",
+ "once": "^1.3.1",
+ "simple-concat": "^1.0.0"
+ }
+ },
+ "node_modules/statuses": {
+ "version": "2.0.2",
+ "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
+ "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/string_decoder": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
+ "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
+ "license": "MIT",
+ "dependencies": {
+ "safe-buffer": "~5.2.0"
+ }
+ },
+ "node_modules/strip-json-comments": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
+ "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/strnum": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/strnum/-/strnum-1.1.2.tgz",
+ "integrity": "sha512-vrN+B7DBIoTTZjnPNewwhx6cBA/H+IS7rfW68n7XxC1y7uoiGQBxaKzqucGUgavX15dJgiGztLJ8vxuEzwqBdA==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/NaturalIntelligence"
+ }
+ ],
+ "license": "MIT"
+ },
+ "node_modules/tar-fs": {
+ "version": "2.1.5",
+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",
+ "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",
+ "license": "MIT",
+ "dependencies": {
+ "chownr": "^1.1.1",
+ "mkdirp-classic": "^0.5.2",
+ "pump": "^3.0.0",
+ "tar-stream": "^2.1.4"
+ }
+ },
+ "node_modules/tar-stream": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",
+ "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "bl": "^4.0.3",
+ "end-of-stream": "^1.4.1",
+ "fs-constants": "^1.0.0",
+ "inherits": "^2.0.3",
+ "readable-stream": "^3.1.1"
+ },
+ "engines": {
+ "node": ">=6"
+ }
+ },
+ "node_modules/toidentifier": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
+ "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.6"
+ }
+ },
+ "node_modules/tunnel-agent": {
+ "version": "0.6.0",
+ "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
+ "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "safe-buffer": "^5.0.1"
+ },
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/type-is": {
+ "version": "1.6.18",
+ "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
+ "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
+ "license": "MIT",
+ "dependencies": {
+ "media-typer": "0.3.0",
+ "mime-types": "~2.1.24"
+ },
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/unpipe": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
+ "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/util-deprecate": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
+ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
+ "license": "MIT"
+ },
+ "node_modules/utils-merge": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
+ "integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.4.0"
+ }
+ },
+ "node_modules/vary": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
+ "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/wrappy": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
+ "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
+ "license": "ISC"
+ }
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,39 @@
@@ -0,0 +1,39 @@
+{
+ "name": "openvibe-sources",
+ "version": "0.1.0",
+ "private": true,
+ "description": "OpenVibe.Sources: the source registry and the ingestion workers (RSS/Atom, sitemaps, JSON-LD, official APIs) that record provenance, retrieval time, content hashes and explicit failure states, never invented content.",
+ "main": "server/index.js",
+ "scripts": {
+ "start": "node server/index.js",
+ "dev": "NODE_ENV=development node server/index.js",
+ "test": "node test/run.js",
+ "seed": "node scripts/seed.js"
+ },
+ "keywords": [
+ "openvibe",
+ "sources",
+ "rss",
+ "atom",
+ "sitemap",
+ "json-ld",
+ "ingestion"
+ ],
+ "author": "OpenVibers",
+ "license": "AGPL-3.0-only",
+ "repository": "github:OpenVibers/OpenVibe.Sources",
+ "dependencies": {
+ "better-sqlite3": "^11.10.0",
+ "dotenv": "^16.4.7",
+ "express": "^4.21.2",
+ "fast-xml-parser": "^4.5.3",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.7.0"
+ },
+ "engines": {
+ "node": ">=22.0.0"
+ },
+ "devDependencies": {
+ "ajv": "^8.20.0",
+ "ajv-formats": "^3.0.1"
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,38 @@
@@ -0,0 +1,38 @@
+#!/usr/bin/env node
+'use strict';
+/**
+ * Load seeds/sources.json into the registry: creates sources that do not exist yet, DISABLED.
+ * Never overwrites or enables an existing source. Enable one deliberately after verifying its terms:
+ * PATCH /api/v1/sources/:key { "enabled": true } (sources.source.manage).
+ *
+ * node scripts/seed.js [--db ./data/sources.db]
+ */
+require('dotenv').config();
+const path = require('path');
+const { load } = require('../server/config');
+const { openDb } = require('../server/db');
+const { createRegistry } = require('../server/registry');
+
+function seed(db, { now = () => Date.now(), by = 'seed' } = {}) {
+ const seeds = require(path.join(__dirname, '..', 'seeds', 'sources.json')).sources;
+ const registry = createRegistry({ db, now });
+ const out = { created: [], kept: [] };
+ for (const s of seeds) {
+ if (registry.get(s.key)) { out.kept.push(s.key); continue; }
+ registry.create({ ...s, enabled: false }, by);
+ out.created.push(s.key);
+ }
+ return out;
+}
+
+if (require.main === module) {
+ const args = process.argv.slice(2);
+ const i = args.indexOf('--db');
+ const config = load();
+ const db = openDb(i >= 0 ? args[i + 1] : config.dbPath);
+ const r = seed(db);
+ console.log(`seeded ${r.created.length} source(s)${r.created.length ? `: ${r.created.join(', ')}` : ''}; kept ${r.kept.length} existing. All seeds are disabled.`);
+ db.close();
+}
+
+module.exports = { seed };
new file mode 100644
new file mode 100644
@@ -0,0 +1,113 @@
@@ -0,0 +1,113 @@
+{
+ "_about": "One real source per category, all DISABLED. Robots notes were checked by hand on 2026-09-22; terms must be re-verified by a person before a source is enabled (PATCH enabled: true). Seeding never overwrites or enables an existing source.",
+ "sources": [
+ {
+ "key": "nasa-news-releases",
+ "name": "NASA news releases",
+ "type": "rss",
+ "category": "news",
+ "homepage_url": "https://www.nasa.gov/",
+ "endpoints": ["https://www.nasa.gov/news-release/feed/"],
+ "robots_note": "robots.txt checked 2026-09-22: User-agent * Allow /. The fetcher re-checks it on every run.",
+ "terms_note": "NASA's media usage guidelines say NASA material is generally not subject to copyright in the US; logos, insignia and third-party material are excluded. Store titles, links and short summaries; link back. Re-verify before enabling.",
+ "license_note": "US federal government work in general; per-item credits may restrict reuse.",
+ "min_interval_ms": 60000,
+ "poll_interval_sec": 1800,
+ "max_items": 100,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
+ },
+ {
+ "key": "nodejs-blog",
+ "name": "Node.js blog",
+ "type": "rss",
+ "category": "blog",
+ "homepage_url": "https://nodejs.org/en/blog",
+ "endpoints": ["https://nodejs.org/en/feed/blog.xml"],
+ "robots_note": "robots.txt checked 2026-09-22: only /dist/ and /docs/ are disallowed; the feed path is allowed.",
+ "terms_note": "Content of the OpenJS Foundation and contributors; read the site's terms before enabling. Titles, links and short summaries only.",
+ "min_interval_ms": 60000,
+ "poll_interval_sec": 3600,
+ "max_items": 50,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
+ },
+ {
+ "key": "steam-reviews-portal-2",
+ "name": "Steam user review signals: Portal 2 (app 620)",
+ "type": "api",
+ "category": "reviews",
+ "homepage_url": "https://store.steampowered.com/app/620/",
+ "endpoints": [{
+ "url": "https://store.steampowered.com/appreviews/620?json=1&filter=recent&language=english&purchase_type=all&num_per_page=20",
+ "format": "json",
+ "items_path": "reviews",
+ "item_kind": "review_signal",
+ "fields": { "identity": "recommendationid", "published_at_epoch": "timestamp_created", "updated_at_epoch": "timestamp_updated" },
+ "extra": {
+ "voted_up": "voted_up", "votes_up": "votes_up", "votes_funny": "votes_funny", "weighted_vote_score": "weighted_vote_score",
+ "playtime_at_review_min": "author.playtime_at_review", "language": "language", "steam_purchase": "steam_purchase",
+ "received_for_free": "received_for_free", "written_during_early_access": "written_during_early_access"
+ }
+ }],
+ "robots_note": "robots.txt checked 2026-09-22: /appreviews is not disallowed.",
+ "terms_note": "Documented store endpoint (partner.steamgames.com/doc/store/getreviews); Steam's terms apply. Review text belongs to its authors and is deliberately not mapped: signals only (recommendation, votes, playtime, dates). Verify terms before enabling.",
+ "min_interval_ms": 60000,
+ "poll_interval_sec": 21600,
+ "max_items": 20,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
+ },
+ {
+ "key": "dealnews-daily",
+ "name": "DealNews daily deals",
+ "type": "rss",
+ "category": "deals",
+ "homepage_url": "https://www.dealnews.com/",
+ "endpoints": ["https://www.dealnews.com/?rss=1"],
+ "robots_note": "robots.txt checked 2026-09-22: Crawl-delay 2 (honoured); the feed URL is not disallowed.",
+ "terms_note": "Copyright DealNews, all rights reserved (feed <copyright>). Headlines, prices exactly as stated and links only; a price is an observation at retrieval time, never a current price. Verify terms before enabling.",
+ "min_interval_ms": 60000,
+ "poll_interval_sec": 3600,
+ "max_items": 100,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
+ },
+ {
+ "key": "staff-coupon-codes",
+ "name": "Coupon codes published by merchants, entered by staff",
+ "type": "manual",
+ "category": "coupons",
+ "terms_note": "Only codes a merchant published itself, entered with the URL of that page as evidence. Validity and expiry stay unknown unless the merchant states them; no code is 'working' because it was entered.",
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
+ },
+ {
+ "key": "sec-xbrl-filings",
+ "name": "SEC EDGAR: latest XBRL filings",
+ "type": "rss",
+ "category": "trade",
+ "homepage_url": "https://www.sec.gov/structureddata/rss-feeds",
+ "endpoints": ["https://www.sec.gov/Archives/edgar/xbrlrss.all.xml"],
+ "robots_note": "robots.txt checked 2026-09-22: /Archives/edgar is allowed; /cgi-bin is disallowed, so the EDGAR 'getcurrent' Atom feed is deliberately not used.",
+ "terms_note": "Public EDGAR data. SEC fair-access policy: a User-Agent that names the requester and a contact (ours does) and at most 10 requests per second. Informational context only — no advice, no execution.",
+ "min_interval_ms": 10000,
+ "poll_interval_sec": 900,
+ "max_items": 200,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "financial",
+ "default_indexability": "noindex"
+ }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,94 @@
@@ -0,0 +1,94 @@
+'use strict';
+/**
+ * Official APIs (JSON or XML) described by a mapping in the source registry — no code per
+ * provider. Each endpoint says where the records are and which field is which:
+ *
+ * { "url": "https://…", "format": "json", "item_kind": "offer", "items_path": "deals",
+ * "fields": { "identity": "dealID", "url": "link", "title": "title", "summary": "…",
+ * "author": "…", "published_at": "…", "published_at_epoch": "releaseDate",
+ * "updated_at": "…", "updated_at_epoch": "lastChange" },
+ * "extra": { "sale_price": "salePrice", "normal_price": "normalPrice", "store_id": "storeID" } }
+ *
+ * Paths are dot-separated keys (array indexes allowed); `items_path` "" means the body itself is
+ * the array. `extra` values are copied as the API gave them (scalars only, strings capped):
+ * a missing field is null, never a default. A record without an identity is skipped.
+ */
+const { parseXml, toText, toIsoDate, epochSecondsToIso, canonicalUrl, toId, ParseError } = require('./util');
+
+const PARSER_VERSION = 'api@1';
+const PATH_RE = /^[A-Za-z0-9_@:$-]+(\.[A-Za-z0-9_@:$-]+)*$/;
+
+function at(obj, path) {
+ if (!path) return obj;
+ let cur = obj;
+ for (const seg of path.split('.')) {
+ if (cur == null || typeof cur !== 'object') return undefined;
+ cur = cur[seg];
+ }
+ return cur;
+}
+
+function scalar(v) {
+ if (v == null) return null;
+ if (typeof v === 'number' || typeof v === 'boolean') return Number.isFinite(v) || typeof v === 'boolean' ? v : null;
+ if (typeof v === 'string') return v.length > 500 ? `${v.slice(0, 499)}…` : v;
+ if (typeof v === 'object' && v['#text'] != null) return scalar(v['#text']);
+ return null;
+}
+
+/** Validate an endpoint mapping (registry side). Returns an error string or null. */
+function checkMapping(ep) {
+ if (!['json', 'xml'].includes(ep.format || 'json')) return 'format must be json or xml';
+ if (ep.items_path !== undefined && ep.items_path !== '' && !PATH_RE.test(ep.items_path)) return 'items_path is malformed';
+ const fields = ep.fields || {};
+ if (!fields.identity && !fields.url) return 'fields.identity or fields.url is required';
+ for (const [k, p] of Object.entries(fields)) {
+ if (!['identity', 'url', 'title', 'summary', 'author', 'published_at', 'published_at_epoch', 'updated_at', 'updated_at_epoch'].includes(k)) return `unknown field ${k}`;
+ if (typeof p !== 'string' || !PATH_RE.test(p)) return `fields.${k} path is malformed`;
+ }
+ const extra = ep.extra || {};
+ if (Object.keys(extra).length > 30) return 'at most 30 extra fields';
+ for (const [k, p] of Object.entries(extra)) {
+ if (!/^[a-z][a-z0-9_]{0,39}$/.test(k) || typeof p !== 'string' || !PATH_RE.test(p)) return `extra.${k} is malformed`;
+ }
+ if (ep.item_kind !== undefined && !/^[a-z][a-z0-9_]{1,39}$/.test(ep.item_kind)) return 'item_kind is malformed';
+ return null;
+}
+
+function parse(text, { url, endpoint = {}, summaryMax = 1000, maxItems = 500 } = {}) {
+ const format = endpoint.format || 'json';
+ let body;
+ if (format === 'xml') body = parseXml(text);
+ else {
+ try { body = JSON.parse(text); } catch { throw new ParseError('body is not valid JSON'); }
+ }
+ const list = at(body, endpoint.items_path || '');
+ if (!Array.isArray(list)) throw new ParseError(`items_path "${endpoint.items_path || ''}" is not an array in the response`);
+ const f = endpoint.fields || {};
+ const items = [];
+ let skipped = 0;
+ const seen = new Set();
+ for (const rec of list) {
+ if (items.length >= maxItems || !rec || typeof rec !== 'object') { skipped++; continue; }
+ const link = f.url ? canonicalUrl(scalar(at(rec, f.url)), url) : null;
+ const identity = (f.identity ? toId(scalar(at(rec, f.identity))) : null) || link;
+ if (!identity || seen.has(identity)) { skipped++; continue; }
+ seen.add(identity);
+ const extra = {};
+ for (const [k, p] of Object.entries(endpoint.extra || {})) extra[k] = scalar(at(rec, p));
+ items.push({
+ identity: String(identity),
+ kind: endpoint.item_kind || 'record',
+ canonical_url: link,
+ title: f.title ? toText(scalar(at(rec, f.title)), 500) : null,
+ summary: f.summary ? toText(scalar(at(rec, f.summary)), summaryMax) : null,
+ authors: f.author ? [toText(scalar(at(rec, f.author)), 200)].filter(Boolean) : [],
+ published_at: f.published_at ? toIsoDate(scalar(at(rec, f.published_at))) : f.published_at_epoch ? epochSecondsToIso(scalar(at(rec, f.published_at_epoch))) : null,
+ updated_at: f.updated_at ? toIsoDate(scalar(at(rec, f.updated_at))) : f.updated_at_epoch ? epochSecondsToIso(scalar(at(rec, f.updated_at_epoch))) : null,
+ fields: extra,
+ });
+ }
+ return { items, skipped, meta: { format } };
+}
+
+module.exports = { parse, PARSER_VERSION, checkMapping };
new file mode 100644
new file mode 100644
@@ -0,0 +1,121 @@
@@ -0,0 +1,121 @@
+'use strict';
+/**
+ * RSS 2.0, RSS 1.0 (RDF) and Atom 1.0 → items of kind `article`.
+ *
+ * Identity: the entry's guid/id, else its link. An entry with neither is skipped (counted), never
+ * given a made-up identity. Summary is text only (HTML stripped) and capped: Sources keeps
+ * metadata and short descriptions, not republishable full text.
+ */
+const { parseXml, toText, toIsoDate, canonicalUrl, toId, asArray, pick, ParseError } = require('./util');
+
+const PARSER_VERSION = 'feed@1';
+
+function atomLink(entry, base) {
+ const links = asArray(entry.link);
+ const alt = links.find(l => typeof l === 'object' && (!l['@_rel'] || l['@_rel'] === 'alternate') && l['@_href'])
+ || links.find(l => typeof l === 'object' && l['@_href'])
+ || links.find(l => typeof l === 'string');
+ if (!alt) return null;
+ return canonicalUrl(typeof alt === 'string' ? alt : alt['@_href'], base);
+}
+
+function authorsOf(list) {
+ return asArray(list).map((a) => {
+ if (a == null) return null;
+ if (typeof a === 'object') return toText(a.name ?? a['#text'] ?? a.__cdata, 200);
+ return toText(a, 200);
+ }).filter(Boolean).slice(0, 20);
+}
+
+function categoriesOf(list) {
+ return asArray(list).map((c) => {
+ if (c == null) return null;
+ if (typeof c === 'object') return toText(c['@_term'] ?? c['#text'] ?? c.__cdata, 100);
+ return toText(c, 100);
+ }).filter(Boolean).slice(0, 30);
+}
+
+function rssItem(it, base, summaryMax) {
+ const guidNode = Array.isArray(it.guid) ? it.guid[0] : it.guid;
+ const guid = toId(guidNode);
+ const link = canonicalUrl(pick(it, 'link'), base);
+ // RSS: a guid is a permalink unless isPermaLink="false".
+ const guidIsPermalink = !(guidNode && typeof guidNode === 'object' && guidNode['@_isPermaLink'] === 'false');
+ const url = link || (guid && guidIsPermalink ? canonicalUrl(guid) : null);
+ const identity = guid || url;
+ if (!identity) return null;
+ const enclosure = asArray(it.enclosure).find(e => e && typeof e === 'object' && e['@_url']);
+ return {
+ identity,
+ kind: 'article',
+ canonical_url: url,
+ title: toText(pick(it, 'title'), 500),
+ summary: toText(pick(it, 'description', 'content:encoded'), summaryMax),
+ authors: authorsOf(it['dc:creator'] ?? it.author),
+ published_at: toIsoDate(pick(it, 'pubDate', 'dc:date')),
+ updated_at: toIsoDate(pick(it, 'atom:updated', 'dc:modified')),
+ fields: {
+ categories: categoriesOf(it.category ?? it['dc:subject']),
+ enclosure: enclosure ? { url: canonicalUrl(enclosure['@_url'], base), type: toId(enclosure['@_type'], 100), length: /^\d{1,15}$/.test(String(enclosure['@_length'] || '')) ? Number(enclosure['@_length']) : null } : null,
+ comments_url: canonicalUrl(pick(it, 'comments'), base),
+ },
+ };
+}
+
+function atomEntry(e, base, summaryMax) {
+ const id = toId(pick(e, 'id'));
+ const url = atomLink(e, base);
+ const identity = id || url;
+ if (!identity) return null;
+ return {
+ identity,
+ kind: 'article',
+ canonical_url: url,
+ title: toText(pick(e, 'title'), 500),
+ summary: toText(pick(e, 'summary', 'content'), summaryMax),
+ authors: authorsOf(e.author),
+ published_at: toIsoDate(pick(e, 'published', 'issued')),
+ updated_at: toIsoDate(pick(e, 'updated', 'modified')),
+ fields: { categories: categoriesOf(e.category) },
+ };
+}
+
+/** parse(text, { url, summaryMax, maxItems }) → { items, skipped, meta } or throws ParseError. */
+function parse(text, { url, summaryMax = 1000, maxItems = 500 } = {}) {
+ const doc = parseXml(text);
+ let raw = [];
+ let meta = {};
+ let make;
+ if (doc.rss && doc.rss.channel) {
+ const ch = Array.isArray(doc.rss.channel) ? doc.rss.channel[0] : doc.rss.channel;
+ raw = asArray(ch.item);
+ meta = { format: 'rss2', title: toText(pick(ch, 'title'), 300), link: canonicalUrl(pick(ch, 'link'), url) };
+ make = (it) => rssItem(it, url, summaryMax);
+ } else if (doc['rdf:RDF']) {
+ const rdf = doc['rdf:RDF'];
+ raw = asArray(rdf.item);
+ const ch = Array.isArray(rdf.channel) ? rdf.channel[0] : rdf.channel;
+ meta = { format: 'rss1', title: ch ? toText(pick(ch, 'title'), 300) : null };
+ make = (it) => rssItem({ ...it, guid: it['@_rdf:about'] ? { '#text': it['@_rdf:about'] } : undefined }, url, summaryMax);
+ } else if (doc.feed) {
+ const feed = doc.feed;
+ raw = asArray(feed.entry);
+ meta = { format: 'atom', title: toText(pick(feed, 'title'), 300), link: atomLink(feed, url) };
+ make = (e) => atomEntry(e, url, summaryMax);
+ } else {
+ throw new ParseError('not an RSS or Atom document');
+ }
+ const items = [];
+ let skipped = 0;
+ const seen = new Set();
+ for (const r of raw) {
+ if (items.length >= maxItems) { skipped++; continue; }
+ const it = r && typeof r === 'object' ? make(r) : null;
+ if (!it || seen.has(it.identity)) { skipped++; continue; }
+ seen.add(it.identity);
+ items.push(it);
+ }
+ return { items, skipped, meta };
+}
+
+module.exports = { parse, PARSER_VERSION };
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+'use strict';
+/**
+ * Adapter registry: source type → parser. `manual` sources have no fetcher; their items are
+ * entered through the API with the evidence URL they came from.
+ */
+const feed = require('./feed');
+const sitemap = require('./sitemap');
+const jsonld = require('./jsonld');
+const api = require('./api');
+
+const ADAPTERS = {
+ rss: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },
+ atom: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/atom+xml, application/rss+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },
+ sitemap: { parse: sitemap.parse, version: sitemap.PARSER_VERSION, accept: 'application/xml, text/xml;q=0.9, */*;q=0.1' },
+ jsonld: { parse: jsonld.parse, version: jsonld.PARSER_VERSION, accept: 'text/html, application/xhtml+xml;q=0.9, */*;q=0.1' },
+ api: { parse: api.parse, version: api.PARSER_VERSION, accept: 'application/json, application/xml;q=0.9, */*;q=0.1' },
+};
+
+const MANUAL_VERSION = 'manual@1';
+
+module.exports = { ADAPTERS, MANUAL_VERSION, checkApiMapping: api.checkMapping };
new file mode 100644
new file mode 100644
@@ -0,0 +1,193 @@
@@ -0,0 +1,193 @@
+'use strict';
+/**
+ * JSON-LD (schema.org) embedded in HTML pages → items of kind product | offer | review | article.
+ *
+ * Types read: Product, Offer, AggregateOffer, Review, NewsArticle, BlogPosting (and Article and
+ * its news subtypes). Values are copied as the page states them: prices stay the strings the page
+ * gave, ratings are the page's own numbers, availability is the schema.org term. Nothing is
+ * computed, averaged or defaulted; a missing price is null. Embedded reviews keep author, rating
+ * and date only (review text belongs to its author).
+ *
+ * A page without any readable JSON-LD block is a parse error: the source promised structured data
+ * and did not deliver it, and the run must say so instead of producing nothing silently.
+ */
+const crypto = require('crypto');
+const { toText, toIsoDate, canonicalUrl, toId, asArray, ParseError } = require('./util');
+
+const PARSER_VERSION = 'jsonld@1';
+const ARTICLE_TYPES = new Set(['NewsArticle', 'BlogPosting', 'Article', 'ReportageNewsArticle', 'AnalysisNewsArticle', 'OpinionNewsArticle', 'BackgroundNewsArticle']);
+const KIND = { Product: 'product', Offer: 'offer', AggregateOffer: 'offer', Review: 'review' };
+
+function typesOf(node) {
+ return asArray(node && node['@type']).map(t => String(t).replace(/^https?:\/\/schema\.org\//, ''));
+}
+
+function kindOf(node) {
+ for (const t of typesOf(node)) {
+ if (KIND[t]) return { kind: KIND[t], type: t };
+ if (ARTICLE_TYPES.has(t)) return { kind: 'article', type: t };
+ }
+ return null;
+}
+
+function extractBlocks(html) {
+ const blocks = [];
+ const re = /<script\b[^>]*type\s*=\s*["']?application\/ld\+json[^>]*>([\s\S]*?)<\/script\s*>/gi;
+ let m;
+ while ((m = re.exec(html)) && blocks.length < 50) blocks.push(m[1]);
+ return blocks;
+}
+
+function flatten(value, out, depth = 0) {
+ if (depth > 3 || value == null) return;
+ if (Array.isArray(value)) { for (const v of value) flatten(v, out, depth + 1); return; }
+ if (typeof value !== 'object') return;
+ if (value['@graph']) flatten(value['@graph'], out, depth + 1);
+ if (value['@type']) out.push(value);
+}
+
+const str = (v, max = 200) => (v == null || typeof v === 'object' ? null : toText(String(v), max));
+const nameOf = (v) => (v == null ? null : typeof v === 'object' ? str(Array.isArray(v) ? (v[0] && v[0].name) : v.name) : str(v));
+/** A number the page stated as a number or numeric string; anything else → null. */
+const numeric = (v) => {
+ if (typeof v === 'number' && Number.isFinite(v)) return v;
+ if (typeof v === 'string' && /^-?\d+(\.\d+)?$/.test(v.trim())) return Number(v.trim());
+ return null;
+};
+/** A price exactly as written, if it is a plain decimal; else null. */
+const priceOf = (v) => {
+ if (typeof v === 'number' && Number.isFinite(v)) return String(v);
+ if (typeof v === 'string' && /^\d+(\.\d+)?$/.test(v.trim())) return v.trim();
+ return null;
+};
+const term = (v) => (typeof v === 'string' ? v.replace(/^https?:\/\/schema\.org\//, '').slice(0, 60) : null);
+const imageOf = (v, base) => {
+ const first = asArray(v)[0];
+ return canonicalUrl(first && typeof first === 'object' ? first.url || first.contentUrl : first, base);
+};
+
+function offerFields(o, base) {
+ if (!o || typeof o !== 'object') return null;
+ return {
+ type: typesOf(o)[0] || null,
+ price: priceOf(o.price),
+ low_price: priceOf(o.lowPrice),
+ high_price: priceOf(o.highPrice),
+ offer_count: numeric(o.offerCount),
+ currency: typeof o.priceCurrency === 'string' && /^[A-Z]{3}$/.test(o.priceCurrency) ? o.priceCurrency : null,
+ availability: term(o.availability),
+ condition: term(o.itemCondition),
+ valid_until: toIsoDate(o.priceValidUntil),
+ url: canonicalUrl(o.url, base),
+ seller: nameOf(o.seller),
+ };
+}
+
+function ratingFields(r) {
+ if (!r || typeof r !== 'object') return null;
+ const out = {
+ value: numeric(r.ratingValue), best: numeric(r.bestRating), worst: numeric(r.worstRating),
+ count: numeric(r.ratingCount), review_count: numeric(r.reviewCount),
+ };
+ return out.value == null && out.count == null && out.review_count == null ? null : out;
+}
+
+function identityFor(node, url, pageUrl, type, ...keys) {
+ const id = typeof node['@id'] === 'string' ? canonicalUrl(node['@id'], pageUrl) || toId(node['@id']) : null;
+ if (id) return id;
+ if (url && url !== pageUrl) return url;
+ const key = keys.find(k => k);
+ if (key) return `${pageUrl}#${type}:${crypto.createHash('sha256').update(String(key)).digest('hex').slice(0, 16)}`;
+ return null;
+}
+
+function toItem(node, pageUrl, summaryMax) {
+ const k = kindOf(node);
+ if (!k) return null;
+ const url = canonicalUrl(node.url, pageUrl);
+ if (k.kind === 'product') {
+ const sku = toId(node.sku, 100);
+ const gtin = toId(node.gtin13 || node.gtin12 || node.gtin14 || node.gtin8 || node.gtin, 20);
+ const name = str(node.name, 500);
+ const identity = identityFor(node, url, pageUrl, 'Product', sku, gtin, name);
+ if (!identity) return null;
+ return {
+ identity, kind: 'product', canonical_url: url || pageUrl,
+ title: name, summary: toText(node.description, summaryMax), authors: [],
+ published_at: null, updated_at: null,
+ fields: {
+ schema_type: k.type, sku, gtin, mpn: toId(node.mpn, 100), brand: nameOf(node.brand),
+ image: imageOf(node.image, pageUrl),
+ offers: asArray(node.offers).map(o => offerFields(o, pageUrl)).filter(Boolean).slice(0, 20),
+ aggregate_rating: ratingFields(node.aggregateRating),
+ reviews: asArray(node.review).slice(0, 10).map(r => (r && typeof r === 'object' ? {
+ author: nameOf(r.author), rating: ratingFields(r.reviewRating), date: toIsoDate(r.datePublished),
+ } : null)).filter(Boolean),
+ },
+ };
+ }
+ if (k.kind === 'offer') {
+ const f = offerFields(node, pageUrl);
+ const itemName = nameOf(node.itemOffered) || str(node.name, 500);
+ const identity = identityFor(node, url, pageUrl, k.type, toId(node.sku, 100), itemName && `${itemName}|${f.price}|${f.currency}`);
+ if (!identity) return null;
+ return {
+ identity, kind: 'offer', canonical_url: url || pageUrl, title: itemName, summary: toText(node.description, summaryMax),
+ authors: [], published_at: null, updated_at: null, fields: { schema_type: k.type, ...f },
+ };
+ }
+ if (k.kind === 'review') {
+ const title = str(node.name, 500) || (nameOf(node.itemReviewed) ? `Review: ${nameOf(node.itemReviewed)}` : null);
+ const authors = asArray(node.author).map(nameOf).filter(Boolean).slice(0, 10);
+ const identity = identityFor(node, url, pageUrl, 'Review', `${nameOf(node.itemReviewed)}|${authors.join(',')}|${node.datePublished || ''}`);
+ if (!identity) return null;
+ return {
+ identity, kind: 'review', canonical_url: url || pageUrl, title,
+ summary: toText(node.reviewBody || node.description, summaryMax), authors,
+ published_at: toIsoDate(node.datePublished), updated_at: toIsoDate(node.dateModified),
+ fields: {
+ schema_type: 'Review', item_reviewed: nameOf(node.itemReviewed),
+ item_reviewed_type: node.itemReviewed && typeof node.itemReviewed === 'object' ? typesOf(node.itemReviewed)[0] || null : null,
+ rating: ratingFields(node.reviewRating), publisher: nameOf(node.publisher),
+ },
+ };
+ }
+ // article
+ const headline = str(node.headline || node.name, 500);
+ const identity = identityFor(node, url, pageUrl, k.type, headline);
+ if (!identity) return null;
+ return {
+ identity, kind: 'article', canonical_url: url || pageUrl, title: headline,
+ summary: toText(node.description, summaryMax),
+ authors: asArray(node.author).map(nameOf).filter(Boolean).slice(0, 20),
+ published_at: toIsoDate(node.datePublished), updated_at: toIsoDate(node.dateModified),
+ fields: {
+ schema_type: k.type, publisher: nameOf(node.publisher), section: str(node.articleSection, 100),
+ image: imageOf(node.image, pageUrl), language: str(node.inLanguage, 20),
+ },
+ };
+}
+
+function parse(text, { url, summaryMax = 1000, maxItems = 500 } = {}) {
+ const blocks = extractBlocks(text);
+ if (!blocks.length) throw new ParseError('no application/ld+json block on the page');
+ const nodes = [];
+ let bad = 0;
+ for (const b of blocks) {
+ try { flatten(JSON.parse(b.trim()), nodes); } catch { bad++; }
+ }
+ if (bad === blocks.length) throw new ParseError('no JSON-LD block is valid JSON');
+ const items = [];
+ let skipped = bad;
+ const seen = new Set();
+ for (const n of nodes) {
+ if (!kindOf(n)) continue; // Organization, BreadcrumbList… are not items
+ const it = toItem(n, url, summaryMax);
+ if (!it || seen.has(it.identity) || items.length >= maxItems) { skipped++; continue; }
+ seen.add(it.identity);
+ items.push(it);
+ }
+ return { items, skipped, meta: { format: 'jsonld', blocks: blocks.length } };
+}
+
+module.exports = { parse, PARSER_VERSION, extractBlocks };
new file mode 100644
new file mode 100644
@@ -0,0 +1,60 @@
@@ -0,0 +1,60 @@
+'use strict';
+/**
+ * Sitemaps (sitemaps.org 0.9): a <urlset> → items of kind `url` (loc, lastmod, plus news:news
+ * title/date when the Google News extension is present); a <sitemapindex> → items of kind
+ * `sitemap` (child sitemaps are listed, not followed: register a child as its own source to
+ * ingest it, so every request stays visible and rate-limited).
+ * Sitemap entries carry no body; only what the file states is stored.
+ */
+const { parseXml, toText, toIsoDate, canonicalUrl, asArray, pick, ParseError } = require('./util');
+
+const PARSER_VERSION = 'sitemap@1';
+
+function parse(text, { url, maxItems = 500 } = {}) {
+ const doc = parseXml(text);
+ const items = [];
+ let skipped = 0;
+ const seen = new Set();
+ const push = (it) => {
+ if (!it || seen.has(it.identity) || items.length >= maxItems) { skipped++; return; }
+ seen.add(it.identity);
+ items.push(it);
+ };
+ if (doc.urlset) {
+ for (const u of asArray(doc.urlset.url)) {
+ const loc = u && typeof u === 'object' ? canonicalUrl(pick(u, 'loc')) : null;
+ if (!loc) { skipped++; continue; }
+ const news = u['news:news'] && (Array.isArray(u['news:news']) ? u['news:news'][0] : u['news:news']);
+ push({
+ identity: loc,
+ kind: 'url',
+ canonical_url: loc,
+ title: news ? toText(pick(news, 'news:title'), 500) : null,
+ summary: null,
+ authors: [],
+ published_at: news ? toIsoDate(pick(news, 'news:publication_date')) : null,
+ updated_at: toIsoDate(pick(u, 'lastmod')),
+ fields: {
+ changefreq: toText(pick(u, 'changefreq'), 20),
+ priority: /^(0(\.\d+)?|1(\.0+)?)$/.test(String(pick(u, 'priority') ?? '')) ? Number(pick(u, 'priority')) : null,
+ publication: news && news['news:publication'] ? toText(pick(news['news:publication'], 'news:name'), 200) : null,
+ },
+ });
+ }
+ return { items, skipped, meta: { format: 'urlset' } };
+ }
+ if (doc.sitemapindex) {
+ for (const s of asArray(doc.sitemapindex.sitemap)) {
+ const loc = s && typeof s === 'object' ? canonicalUrl(pick(s, 'loc')) : null;
+ if (!loc) { skipped++; continue; }
+ push({
+ identity: loc, kind: 'sitemap', canonical_url: loc, title: null, summary: null, authors: [],
+ published_at: null, updated_at: toIsoDate(pick(s, 'lastmod')), fields: {},
+ });
+ }
+ return { items, skipped, meta: { format: 'sitemapindex', base: url } };
+ }
+ throw new ParseError('not a sitemap (<urlset> or <sitemapindex>)');
+}
+
+module.exports = { parse, PARSER_VERSION };
new file mode 100644
new file mode 100644
@@ -0,0 +1,167 @@
@@ -0,0 +1,167 @@
+'use strict';
+/**
+ * Helpers every adapter shares. Adapters turn a fetched body into parsed items and nothing
+ * else: a field the source did not provide stays null — never guessed, defaulted or invented.
+ */
+const crypto = require('crypto');
+const { XMLParser } = require('fast-xml-parser');
+
+class ParseError extends Error {
+ constructor(detail) { super(detail); this.code = 'parse_error'; }
+}
+
+function sha256(buf) {
+ return crypto.createHash('sha256').update(buf).digest('hex');
+}
+
+function canonicalJson(value) {
+ if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`;
+ if (value && typeof value === 'object') {
+ return `{${Object.keys(value).sort().filter(k => value[k] !== undefined).map(k => `${JSON.stringify(k)}:${canonicalJson(value[k])}`).join(',')}}`;
+ }
+ return JSON.stringify(value === undefined ? null : value);
+}
+
+/** Decode a body: charset from Content-Type, else the XML declaration / meta tag, else UTF-8. */
+function decodeBody(buf, contentType) {
+ let charset = /charset=["']?([\w-]+)/i.exec(String(contentType || ''))?.[1];
+ if (!charset) {
+ const head = buf.subarray(0, 1024).toString('latin1');
+ charset = /<\?xml[^>]*encoding=["']([\w-]+)["']/i.exec(head)?.[1] || /<meta[^>]+charset=["']?([\w-]+)/i.exec(head)?.[1];
+ }
+ let label = String(charset || 'utf-8').toLowerCase();
+ if (label === 'iso-8859-1' || label === 'latin1') label = 'windows-1252';
+ try {
+ return new TextDecoder(label, { fatal: false }).decode(buf).replace(/^\uFEFF/, '');
+ } catch {
+ return new TextDecoder('utf-8').decode(buf).replace(/^\uFEFF/, '');
+ }
+}
+
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', hellip: '…', mdash: '—', ndash: '–', rsquo: '’', lsquo: '‘', rdquo: '”', ldquo: '“' };
+
+function decodeEntities(s) {
+ return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
+ if (e[0] === '#') {
+ const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
+ return Number.isFinite(code) && code > 0 && code < 0x110000 ? String.fromCodePoint(code) : '';
+ }
+ return ENTITIES[e.toLowerCase()] ?? m;
+ });
+}
+
+/** HTML or text → plain text, whitespace collapsed, capped. Scripts and styles are dropped whole. */
+function toText(value, max = 1000) {
+ if (value == null) return null;
+ // Bounded input: only a short summary survives anyway, and the regexes below stay linear.
+ let s = String(typeof value === 'object' ? (value['#text'] ?? value.__cdata ?? '') : value).slice(0, 100000);
+ // XML entities first (the parser leaves them), then markup, then HTML entities of the markup.
+ s = decodeEntities(String(s))
+ .replace(/<(script|style|noscript)[\s\S]*?<\/\1>/gi, ' ')
+ .replace(/<br\s*\/?>|<\/p>|<\/li>|<\/h\d>/gi, ' ')
+ .replace(/<[^>]*>/g, ' ');
+ s = decodeEntities(s).replace(/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F]/g, '').replace(/\s+/g, ' ').trim();
+ if (!s) return null;
+ return s.length > max ? `${s.slice(0, max - 1).trimEnd()}…` : s;
+}
+
+/** RFC 822 / RFC 3339 / ISO dates → ISO string; anything unparseable → null (never "now"). */
+function toIsoDate(value) {
+ if (value == null) return null;
+ const s = decodeEntities(String(typeof value === 'object' ? value['#text'] ?? '' : value)).trim();
+ if (!s || s.length > 64) return null;
+ // Refuse bare numbers: ambiguous epoch units are guesses.
+ if (/^\d+$/.test(s)) return null;
+ const t = Date.parse(s);
+ if (!Number.isFinite(t)) return null;
+ const year = new Date(t).getUTCFullYear();
+ if (year < 1990 || year > 2200) return null;
+ return new Date(t).toISOString();
+}
+
+/** Epoch seconds (for APIs that document them) → ISO, or null. */
+function epochSecondsToIso(value) {
+ const n = Number(value);
+ if (!Number.isFinite(n) || n <= 0) return null;
+ const d = new Date(n * 1000);
+ const y = d.getUTCFullYear();
+ return y >= 1990 && y <= 2200 ? d.toISOString() : null;
+}
+
+const TRACKING = /^(utm_[a-z]+|fbclid|gclid|mc_cid|mc_eid|_hsenc|_hsmi|mkt_tok)$/i;
+
+/** http(s) URL → canonical form (lowercase host, no fragment, no tracking params, no default port); else null. */
+function canonicalUrl(value, base) {
+ if (value == null) return null;
+ const raw = decodeEntities(String(typeof value === 'object' ? value['@_href'] ?? value['#text'] ?? '' : value)).trim();
+ if (!raw || raw.length > 2048) return null;
+ let url;
+ try { url = base ? new URL(raw, base) : new URL(raw); } catch { return null; }
+ if (url.protocol !== 'http:' && url.protocol !== 'https:') return null;
+ if (url.username || url.password) return null;
+ url.hash = '';
+ url.hostname = url.hostname.toLowerCase();
+ if ((url.protocol === 'https:' && url.port === '443') || (url.protocol === 'http:' && url.port === '80')) url.port = '';
+ for (const k of [...url.searchParams.keys()]) if (TRACKING.test(k)) url.searchParams.delete(k);
+ return url.toString();
+}
+
+/** A plain identifier string (guid, sku…) or null. */
+function toId(value, max = 512) {
+ if (value == null) return null;
+ const s = decodeEntities(String(typeof value === 'object' ? value['#text'] ?? '' : value)).trim();
+ return s && s.length <= max ? s : null;
+}
+
+function asArray(v) {
+ if (v == null) return [];
+ return Array.isArray(v) ? v : [v];
+}
+
+/**
+ * Parse XML safely: a DOCTYPE with entity declarations is refused (entity expansion is how XML
+ * bombs work) and the parser expands no entities at all — the helpers above decode the predefined
+ * and numeric ones per value (large real feeds carry thousands of &, which trips the
+ * parser's own expansion limit).
+ */
+function parseXml(text) {
+ if (/<!DOCTYPE[^>]*\[/i.test(text) || /<!ENTITY/i.test(text)) throw new ParseError('XML with entity declarations is refused');
+ const parser = new XMLParser({
+ ignoreAttributes: false,
+ attributeNamePrefix: '@_',
+ removeNSPrefix: false,
+ processEntities: false,
+ trimValues: true,
+ parseTagValue: false,
+ parseAttributeValue: false,
+ cdataPropName: '__cdata',
+ });
+ try {
+ return parser.parse(text, true);
+ } catch (err) {
+ throw new ParseError(`not well-formed XML: ${String(err.message).split('\n')[0].slice(0, 200)}`);
+ }
+}
+
+/** First non-empty text value of the given keys on an XML node. */
+function pick(node, ...keys) {
+ for (const k of keys) {
+ const v = node && node[k];
+ if (v == null) continue;
+ const first = Array.isArray(v) ? v[0] : v;
+ if (first == null) continue;
+ if (typeof first === 'object') {
+ const t = first['#text'] ?? first.__cdata;
+ if (t != null && String(t).trim()) return t;
+ if (first['@_href']) return first;
+ continue;
+ }
+ if (String(first).trim()) return first;
+ }
+ return null;
+}
+
+module.exports = {
+ ParseError, sha256, canonicalJson, decodeBody, toText, toIsoDate, epochSecondsToIso, canonicalUrl, toId,
+ asArray, parseXml, pick, decodeEntities,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,113 @@
@@ -0,0 +1,113 @@
+'use strict';
+/**
+ * Items API (service tokens).
+ *
+ * sources.item.read
+ * GET /api/v1/items?source=&category=&after=<change_seq>&limit=&include_removed=1
+ * items in change order (created, updated, removed); resume from next_after. Each page
+ * carries the staleness of every source it contains, so a consumer never mistakes an old
+ * observation for a current one.
+ * GET /api/v1/items/:id?revisions=1
+ * sources.source.manage
+ * POST /api/v1/sources/:key/items add or revise an item of a `manual` source (evidence URL required)
+ * DELETE /api/v1/items/:id { reason } — takedown/licence removal, sticky
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { CAPS } = require('../auth');
+const { CATEGORIES } = require('../registry');
+const { MANUAL_VERSION } = require('../adapters');
+const { toText, toIsoDate, canonicalUrl, toId } = require('../adapters/util');
+
+function manualItem(body) {
+ if (!body || typeof body !== 'object' || Array.isArray(body)) return { error: 'body must be an object' };
+ const url = canonicalUrl(body.url);
+ if (!url) return { error: 'url (the evidence: where this was published) must be an http(s) URL' };
+ const title = toText(body.title, 500);
+ if (!title) return { error: 'title is required' };
+ const kind = body.kind === undefined ? 'record' : body.kind;
+ if (!/^[a-z][a-z0-9_]{1,39}$/.test(String(kind))) return { error: 'kind is malformed' };
+ if (body.published_at != null && !toIsoDate(body.published_at)) return { error: 'published_at must be a date the source states' };
+ const fields = body.fields == null ? {} : body.fields;
+ if (typeof fields !== 'object' || Array.isArray(fields) || Object.keys(fields).length > 30) return { error: 'fields must be an object of at most 30 scalars' };
+ for (const [k, v] of Object.entries(fields)) {
+ if (!/^[a-z][a-z0-9_]{0,39}$/.test(k)) return { error: `fields.${k} is malformed` };
+ if (!(v === null || typeof v === 'boolean' || (typeof v === 'number' && Number.isFinite(v)) || (typeof v === 'string' && v.length <= 500))) return { error: `fields.${k} must be a scalar` };
+ }
+ const identity = body.identity == null ? url : toId(body.identity);
+ if (!identity) return { error: 'identity is malformed' };
+ return {
+ item: {
+ identity, kind, canonical_url: url, title, summary: toText(body.summary, 1000),
+ authors: Array.isArray(body.authors) ? body.authors.map(a => toText(a, 200)).filter(Boolean).slice(0, 20) : [],
+ published_at: body.published_at == null ? null : toIsoDate(body.published_at),
+ updated_at: body.updated_at == null ? null : toIsoDate(body.updated_at),
+ fields,
+ },
+ };
+}
+
+function itemsRouter({ db, registry, items, auth, relay, now }) {
+ const router = express.Router();
+ const read = auth.requireCap(CAPS.items);
+ const manage = auth.requireCap(CAPS.manage);
+
+ router.get('/api/v1/items', read, (req, res) => {
+ const source = req.query.source ? String(req.query.source) : null;
+ const category = req.query.category ? String(req.query.category) : null;
+ if (category && !CATEGORIES.includes(category)) return http.sendProblem(res, 400, 'sources.bad_query', { detail: `category must be one of ${CATEGORIES.join('|')}`, ctx: req.ov });
+ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
+ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 100, 1), 500);
+ const page = items.list({ source, category, after, limit, includeRemoved: req.query.include_removed === '1' });
+ const keys = [...new Set(page.items.map(i => i.source_key))];
+ const sources = {};
+ for (const k of keys) {
+ const row = registry.get(k);
+ if (row) {
+ const h = registry.health(row);
+ sources[k] = { status: h.status, stale: h.stale, last_success_at: h.last_success_at };
+ }
+ }
+ res.json({ ...page, sources });
+ });
+
+ router.get('/api/v1/items/:id', read, (req, res) => {
+ const item = items.get(String(req.params.id), { revisions: req.query.revisions === '1' });
+ if (!item) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such item', ctx: req.ov });
+ const row = registry.get(item.source_key);
+ const h = row ? registry.health(row) : null;
+ res.json({ item, source: h ? { key: item.source_key, status: h.status, stale: h.stale, last_success_at: h.last_success_at } : null });
+ });
+
+ router.post('/api/v1/sources/:key/items', manage, (req, res) => {
+ const row = registry.get(req.params.key);
+ if (!row) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such source', ctx: req.ov });
+ if (row.type !== 'manual') return http.sendProblem(res, 409, 'sources.not_manual', { detail: 'items of fetched sources come from their fetches only', ctx: req.ov });
+ const source = registry.fromRow(row);
+ if (!source.enabled) return http.sendProblem(res, 409, 'sources.disabled', { detail: 'the source is disabled', ctx: req.ov });
+ const m = manualItem(req.body);
+ if (m.error) return http.sendProblem(res, 422, 'sources.bad_item', { detail: m.error, ctx: req.ov });
+ const t = now();
+ const counts = db.transaction(() => items.ingest(source, [m.item], { runId: null, rawBodyHash: null, parserVersion: MANUAL_VERSION, retrievedAt: t, enteredBy: req.principal.sub }))();
+ db.prepare('UPDATE sources SET last_success_at = ?, last_run_at = ? WHERE key = ?').run(t, t, source.key);
+ if (relay) relay.flush().catch(() => {});
+ const saved = db.prepare('SELECT id FROM items WHERE source_key = ? AND identity = ?').get(source.key, m.item.identity);
+ const outcome = counts.created ? 'created' : counts.updated ? 'updated' : counts.skipped ? 'removed' : 'unchanged';
+ res.status(counts.created ? 201 : 200).json({ outcome, item: items.get(saved.id) });
+ });
+
+ router.delete('/api/v1/items/:id', manage, (req, res) => {
+ const reason = toText(req.body && req.body.reason, 300);
+ if (!reason) return http.sendProblem(res, 422, 'sources.reason_required', { detail: 'a removal needs a reason (takedown, licence, error…)', ctx: req.ov });
+ const found = items.get(String(req.params.id));
+ if (!found) return http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such item', ctx: req.ov });
+ const source = registry.fromRow(registry.get(found.source_key));
+ const item = items.remove(source, found.id, reason, req.principal.sub);
+ if (relay) relay.flush().catch(() => {});
+ res.json({ item });
+ });
+
+ return router;
+}
+
+module.exports = { itemsRouter, manualItem };
new file mode 100644
new file mode 100644
@@ -0,0 +1,128 @@
@@ -0,0 +1,128 @@
+'use strict';
+/**
+ * Registry and fetch-run API (service tokens, one capability per route).
+ *
+ * sources.source.read
+ * GET /api/v1/sources every source with health and staleness
+ * GET /api/v1/sources/:key
+ * GET /api/v1/sources/:key/runs?before=&limit= fetch runs, newest first
+ * GET /api/v1/runs?state=&source=&after=&limit= fetch runs across sources, oldest first
+ * GET /api/v1/health status counts and every source that is not healthy
+ * sources.source.manage
+ * POST /api/v1/sources create (disabled unless enabled: true + notes)
+ * PATCH /api/v1/sources/:key change fields (enable/disable too)
+ * DELETE /api/v1/sources/:key only while it has no items
+ * POST /api/v1/sources/:key/fetch run now (409 while a run is in flight)
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { CAPS } = require('../auth');
+const { RegistryError } = require('../registry');
+
+const STATES = ['ok', 'not_modified', 'http_error', 'timeout', 'robots_denied', 'parse_error', 'rate_limited', 'disabled'];
+
+function runView(r) {
+ const iso = (v) => (v == null ? null : new Date(v).toISOString());
+ return {
+ id: r.id, source_key: r.source_key, endpoint_url: r.endpoint_url, trigger: r.trigger,
+ started_at: iso(r.started_at), finished_at: iso(r.finished_at), state: r.state, http_status: r.http_status,
+ error_code: r.error_code, detail: r.detail, bytes: r.bytes, raw_body_hash: r.raw_body_hash, parser_version: r.parser_version,
+ items: { seen: r.items_seen, created: r.items_created, updated: r.items_updated, unchanged: r.items_unchanged, skipped: r.items_skipped },
+ seq: r.rid,
+ };
+}
+
+function sourcesRouter({ db, registry, ingest, auth }) {
+ const router = express.Router();
+ const read = auth.requireCap(CAPS.read);
+ const manage = auth.requireCap(CAPS.manage);
+
+ const fail = (res, req, err) => {
+ if (err instanceof RegistryError) {
+ const status = err.code === 'sources.exists' || err.code === 'sources.has_items' ? 409 : 422;
+ return http.sendProblem(res, status, err.code, { detail: err.message, ctx: req.ov });
+ }
+ throw err;
+ };
+ const notFound = (res, req) => http.sendProblem(res, 404, 'sources.not_found', { detail: 'no such source', ctx: req.ov });
+
+ router.get('/api/v1/sources', read, (req, res) => {
+ res.json({ sources: registry.all().map(registry.view) });
+ });
+
+ router.get('/api/v1/sources/:key', read, (req, res) => {
+ const row = registry.get(req.params.key);
+ if (!row) return notFound(res, req);
+ res.json({ source: registry.view(row) });
+ });
+
+ router.get('/api/v1/sources/:key/runs', read, (req, res) => {
+ if (!registry.get(req.params.key)) return notFound(res, req);
+ const before = /^\d{1,15}$/.test(String(req.query.before || '')) ? Number(req.query.before) : Number.MAX_SAFE_INTEGER;
+ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 50, 1), 200);
+ const rows = db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? AND rid < ? ORDER BY rid DESC LIMIT ?').all(req.params.key, before, limit);
+ res.json({ runs: rows.map(runView), next_before: rows.length === limit ? rows[rows.length - 1].rid : null });
+ });
+
+ router.get('/api/v1/runs', read, (req, res) => {
+ const state = req.query.state ? String(req.query.state) : null;
+ if (state && state !== 'failed' && !STATES.includes(state)) return http.sendProblem(res, 400, 'sources.bad_query', { detail: `state must be failed or one of ${STATES.join('|')}`, ctx: req.ov });
+ const after = /^\d{1,15}$/.test(String(req.query.after || '')) ? Number(req.query.after) : 0;
+ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 100, 1), 500);
+ const rows = db.prepare(`SELECT * FROM fetch_runs WHERE rid > @after AND (@source IS NULL OR source_key = @source)
+ AND (@state IS NULL OR state = @state OR (@state = 'failed' AND state IN ('http_error','timeout','robots_denied','parse_error','rate_limited')))
+ ORDER BY rid LIMIT @limit`).all({ after, source: req.query.source ? String(req.query.source) : null, state, limit });
+ res.json({ runs: rows.map(runView), next_after: rows.length ? rows[rows.length - 1].rid : after });
+ });
+
+ router.get('/api/v1/health', read, (req, res) => {
+ const views = registry.all().map(r => ({ key: r.key, category: r.category, type: r.type, ...registry.health(r) }));
+ const counts = {};
+ for (const v of views) counts[v.status] = (counts[v.status] || 0) + 1;
+ res.json({ counts, attention: views.filter(v => ['failing', 'stale', 'never_fetched'].includes(v.status)) });
+ });
+
+ router.post('/api/v1/sources', manage, (req, res) => {
+ try {
+ const row = registry.create(req.body, req.principal.sub);
+ res.status(201).json({ source: registry.view(row) });
+ } catch (err) { fail(res, req, err); }
+ });
+
+ router.patch('/api/v1/sources/:key', manage, (req, res) => {
+ try {
+ const row = registry.patch(req.params.key, req.body, req.principal.sub);
+ if (!row) return notFound(res, req);
+ res.json({ source: registry.view(row) });
+ } catch (err) { fail(res, req, err); }
+ });
+
+ router.delete('/api/v1/sources/:key', manage, (req, res) => {
+ try {
+ const ok = db.transaction(() => {
+ const r = registry.remove(req.params.key);
+ if (r) {
+ db.prepare('DELETE FROM fetch_runs WHERE source_key = ?').run(req.params.key);
+ db.prepare('DELETE FROM endpoint_state WHERE source_key = ?').run(req.params.key);
+ }
+ return r;
+ })();
+ if (!ok) return notFound(res, req);
+ res.status(204).end();
+ } catch (err) { fail(res, req, err); }
+ });
+
+ router.post('/api/v1/sources/:key/fetch', manage, async (req, res, next) => {
+ try {
+ const out = await ingest.run(req.params.key, { trigger: 'manual' });
+ if (!out) return notFound(res, req);
+ if (out.manual) return http.sendProblem(res, 409, 'sources.manual_source', { detail: 'a manual source is not fetched; add items with POST /api/v1/sources/:key/items', ctx: req.ov });
+ if (out.busy) return http.sendProblem(res, 409, 'sources.fetch_in_flight', { detail: 'a run of this source is in flight', ctx: req.ov });
+ res.json({ runs: out.runs.map(r => runView({ ...r, rid: null })) });
+ } catch (err) { next(err); }
+ });
+
+ return router;
+}
+
+module.exports = { sourcesRouter, runView };
new file mode 100644
new file mode 100644
@@ -0,0 +1,80 @@
@@ -0,0 +1,80 @@
+'use strict';
+/** Express app: request context, the v1 API, health/readiness. */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { sourcesRouter } = require('./api/sources');
+const { itemsRouter } = require('./api/items');
+const pkg = require('../package.json');
+
+function createApp({ config, db, registry, items, ingest, scheduler, auth, keys, outbox, relay, now, log = console }) {
+ const app = express();
+ app.disable('x-powered-by');
+ app.set('trust proxy', 'loopback');
+ app.use(http.middleware());
+ app.use((req, res, next) => {
+ res.setHeader('X-Content-Type-Options', 'nosniff');
+ res.setHeader('Cache-Control', 'no-store');
+ next();
+ });
+ app.use('/api', express.json({ limit: '256kb', type: ['application/json', 'application/*+json'] }));
+
+ app.get('/api/health', (_req, res) => {
+ res.json({ status: 'ok', service: 'openvibe-sources', version: pkg.version });
+ });
+
+ app.get('/api/ready', (_req, res) => {
+ let dbOk = false;
+ try { dbOk = db.prepare('SELECT 1 AS ok').get().ok === 1; } catch { dbOk = false; }
+ const checks = { db: dbOk, key: keys.loaded(), worker: !config.worker.enabled || scheduler.running() };
+ const ready = Object.values(checks).every(Boolean);
+ let sources = null;
+ if (dbOk) {
+ sources = {};
+ for (const r of registry.all()) {
+ const s = registry.health(r).status;
+ sources[s] = (sources[s] || 0) + 1;
+ }
+ }
+ res.status(ready ? 200 : 503).json({
+ status: ready ? 'ready' : 'not_ready',
+ checks,
+ sources,
+ runs_in_flight: ingest.inflight().length,
+ outbox: dbOk ? { pending: outbox.pending(), rejected: outbox.rejected(), relay: config.events.url ? (relay.running() ? 'running' : 'stopped') : 'off (EVENTS_URL unset)' } : null,
+ });
+ });
+
+ app.use(sourcesRouter({ db, registry, ingest, auth }));
+ app.use(itemsRouter({ db, registry, items, auth, relay, now }));
+
+ app.get('/', (_req, res) => {
+ res.type('text/plain').send([
+ 'OpenVibe.Sources: the source registry and ingestion workers for the network\'s publication products.',
+ 'Every item carries its source, canonical URL, retrieval time, content hash and terms; a failed fetch is',
+ 'recorded as a failure and never replaced with invented content.',
+ '',
+ 'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',
+ 'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',
+ 'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
+ 'GET /api/health, /api/ready',
+ '',
+ 'Source: https://github.com/OpenVibers/OpenVibe.Sources',
+ '',
+ ].join('\n'));
+ });
+
+ app.use((req, res) => http.sendProblem(res, 404, 'sources.not_found', { detail: `no route ${req.method} ${req.path}`, ctx: req.ov }));
+
+ // eslint-disable-next-line no-unused-vars
+ app.use((err, req, res, _next) => {
+ if (err.type === 'entity.parse.failed') return http.sendProblem(res, 400, 'sources.bad_json', { detail: 'request body is not valid JSON', ctx: req.ov });
+ if (err.type === 'entity.too.large') return http.sendProblem(res, 413, 'sources.too_large', { detail: 'request body too large', ctx: req.ov });
+ log.error(`[app] ${req.method} ${req.path}: ${err.stack || err}`);
+ if (res.headersSent) return res.end();
+ return http.sendProblem(res, 500, 'sources.internal', { detail: 'internal error', ctx: req.ov });
+ });
+
+ return app;
+}
+
+module.exports = { createApp };
new file mode 100644
new file mode 100644
@@ -0,0 +1,104 @@
@@ -0,0 +1,104 @@
+'use strict';
+/**
+ * Service-token authentication and capability checks. Every API route is for services (other
+ * OpenVibe services, or the Network admin surface acting through its own principal): an
+ * RS256 client-credentials token from OpenVibe.Network (audience openvibe.sources), verified
+ * offline, and ONE capability per route.
+ *
+ * The sources.* capabilities are proposed in docs/capabilities-proposal/ and are not in
+ * openvibe-contracts v0.7.0 yet; checkCapability() decides them with the library's own grant rule
+ * (exact id or a `family.*` grant) until a release knows them, then the library decides.
+ */
+const crypto = require('crypto');
+const { serviceAuth, capabilities, http } = require('openvibe-contracts');
+
+const CAPS = Object.freeze({
+ read: 'sources.source.read',
+ items: 'sources.item.read',
+ manage: 'sources.source.manage',
+});
+const PROPOSED = new Set(Object.values(CAPS));
+
+function createKeyStore({ urls = [], pem = null, fetchImpl = globalThis.fetch, log = console } = {}) {
+ let key = pem ? toPem(pem) : null;
+ let retryTimer = null;
+ let refreshTimer = null;
+
+ function toPem(value) {
+ return crypto.createPublicKey(value).export({ type: 'spki', format: 'pem' });
+ }
+
+ async function fetchOnce() {
+ for (const base of urls) {
+ if (!base) continue;
+ const url = `${base}/api/.well-known/jwks`;
+ try {
+ const res = await fetchImpl(url, { signal: AbortSignal.timeout(8000) });
+ if (!res.ok) throw new Error(`HTTP ${res.status}`);
+ const body = await res.json();
+ const jwk = (body.keys || []).find(k => k.kty === 'RSA');
+ if (jwk) key = toPem({ key: jwk, format: 'jwk' });
+ else if (typeof body.public_key === 'string' && body.public_key.includes('BEGIN')) key = toPem(body.public_key);
+ else throw new Error('no RSA key in response');
+ log.log(`[auth] Network public key loaded from ${base}`);
+ return key;
+ } catch (err) {
+ log.warn(`[auth] key fetch from ${url} failed: ${err.message}`);
+ }
+ }
+ return null;
+ }
+
+ function start() {
+ if (pem) return Promise.resolve(key);
+ const attempt = async () => {
+ const k = await fetchOnce();
+ if (!k && !key) {
+ retryTimer = setTimeout(attempt, 30 * 1000);
+ retryTimer.unref?.();
+ }
+ return k;
+ };
+ refreshTimer = setInterval(() => { fetchOnce().catch(() => {}); }, 6 * 60 * 60 * 1000);
+ refreshTimer.unref?.();
+ return attempt();
+ }
+
+ function stop() {
+ clearTimeout(retryTimer);
+ clearInterval(refreshTimer);
+ }
+
+ return { get: () => key, loaded: () => Boolean(key), start, stop };
+}
+
+function checkCapability(claims, capabilityId) {
+ if (!capabilities.get(capabilityId) && PROPOSED.has(capabilityId)) {
+ return capabilities.grants(claims && claims.cap, capabilityId)
+ ? { allowed: true, code: null, reason: null }
+ : { allowed: false, code: 'capability.denied', reason: `${capabilityId} not granted` };
+ }
+ return capabilities.check(claims, capabilityId);
+}
+
+function createAuth({ config, keys }) {
+ /** Express guard: one capability. Sets req.principal = { sub, cap, jti }. */
+ function requireCap(id) {
+ return function capGuard(req, res, next) {
+ const ctx = req.ov;
+ const h = String(req.headers.authorization || '');
+ if (!h.startsWith('Bearer ')) return http.sendProblem(res, 401, 'token.missing', { detail: 'a service token is required', ctx });
+ const publicKey = keys.get();
+ if (!publicKey) return http.sendProblem(res, 503, 'token.unavailable', { detail: 'signing key not loaded yet', ctx });
+ const r = serviceAuth.verifyServiceToken(h.slice(7).trim(), { publicKey, issuer: config.issuer, audience: config.audience });
+ if (!r.ok) return http.sendProblem(res, 401, r.code, { detail: r.reason, ctx });
+ const c = checkCapability(r.claims, id);
+ if (!c.allowed) return http.sendProblem(res, 403, c.code, { detail: c.reason, ctx });
+ req.principal = { sub: r.claims.sub, cap: r.claims.cap, jti: r.claims.jti };
+ return next();
+ };
+ }
+ return { requireCap };
+}
+
+module.exports = { CAPS, PROPOSED, createKeyStore, createAuth, checkCapability };
new file mode 100644
new file mode 100644
@@ -0,0 +1,80 @@
@@ -0,0 +1,80 @@
+'use strict';
+/**
+ * OpenVibe.Sources configuration. Everything comes from the environment (.env in development,
+ * /etc/openvibe/sources.env in production); see .env.example for the documented list.
+ *
+ * load(env) is pure so tests can build a config without touching process.env. `env` is also kept
+ * as config.secrets: a source's credential is read from it by environment-variable NAME at fetch
+ * time and never stored, logged or returned.
+ */
+const pkg = require('../package.json');
+
+const int = (v, d) => {
+ const n = parseInt(v, 10);
+ return Number.isFinite(n) ? n : d;
+};
+const list = (v, d) => (v == null || v === '' ? d : String(v).split(',').map(s => s.trim()).filter(Boolean));
+const strip = (v) => String(v || '').replace(/\/$/, '');
+
+function load(env = process.env) {
+ const nodeEnv = env.NODE_ENV || 'development';
+ const isProduction = nodeEnv === 'production';
+ const port = int(env.PORT, 4720);
+ return {
+ port,
+ host: env.HOST || '127.0.0.1',
+ nodeEnv,
+ isProduction,
+ serviceId: 'sources',
+ baseUrl: strip(env.BASE_URL || (isProduction ? 'https://sources.openvibe.network' : `http://localhost:${port}`)),
+
+ networkUrl: strip(env.OV_NETWORK_URL || 'https://openvibe.network'),
+ networkInternalUrl: strip(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
+ issuer: strip(env.OV_NETWORK_ISSUER || env.OV_NETWORK_URL || 'https://openvibe.network'),
+ networkPublicKey: env.OV_NETWORK_PUBLIC_KEY ? env.OV_NETWORK_PUBLIC_KEY.replace(/\\n/g, '\n') : null,
+ audience: 'openvibe.sources',
+ oauth: {
+ clientId: env.OV_OAUTH_CLIENT_ID || 'sources',
+ clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
+ },
+
+ dbPath: env.SOURCES_DB_PATH || './data/sources.db',
+
+ events: {
+ url: strip(env.EVENTS_URL || ''),
+ relayIntervalMs: int(env.EVENTS_RELAY_INTERVAL_MS, 2000),
+ },
+
+ // Ingestion
+ worker: {
+ enabled: env.SOURCES_WORKER !== 'off',
+ tickMs: int(env.SOURCES_TICK_MS, 5000),
+ maxConcurrent: int(env.SOURCES_MAX_CONCURRENT, 4),
+ // Longest wait between polls of a failing source (exponential backoff cap).
+ maxBackoffMs: int(env.SOURCES_MAX_BACKOFF_MS, 6 * 3600 * 1000),
+ },
+ fetch: {
+ userAgent: env.SOURCES_USER_AGENT || `OpenVibeSources/${pkg.version} (+https://sources.openvibe.network; [email protected])`,
+ // Product token matched against robots.txt User-agent lines.
+ robotsAgent: env.SOURCES_ROBOTS_AGENT || 'OpenVibeSources',
+ timeoutMs: int(env.SOURCES_FETCH_TIMEOUT_MS, 20000),
+ maxBytes: int(env.SOURCES_MAX_BYTES, 5 * 1024 * 1024),
+ maxRedirects: int(env.SOURCES_MAX_REDIRECTS, 5),
+ robotsMaxBytes: 512 * 1024,
+ robotsTtlMs: int(env.SOURCES_ROBOTS_TTL_MS, 24 * 3600 * 1000),
+ // Minimum gap between any two requests to one host, whatever the sources say.
+ hostMinIntervalMs: int(env.SOURCES_HOST_MIN_INTERVAL_MS, 1000),
+ allowedPorts: list(env.SOURCES_ALLOWED_PORTS, ['80', '443']).map(Number),
+ // Exact hostnames/IPs that may resolve to private or loopback addresses (tests and
+ // deliberate on-host sources only). Empty in production.
+ allowPrivateHosts: list(env.SOURCES_ALLOW_PRIVATE_HOSTS, []),
+ },
+ items: {
+ maxPerFetch: int(env.SOURCES_MAX_ITEMS_PER_FETCH, 500),
+ summaryMax: 1000,
+ },
+ secrets: env,
+ };
+}
+
+module.exports = { load };
new file mode 100644
new file mode 100644
@@ -0,0 +1,158 @@
@@ -0,0 +1,158 @@
+'use strict';
+/**
+ * The Sources database (SQLite, WAL). One file owned by this service only; created on boot,
+ * idempotently.
+ */
+const fs = require('fs');
+const path = require('path');
+const Database = require('better-sqlite3');
+const outbox = require('./events/outbox');
+
+const SCHEMA = `
+CREATE TABLE IF NOT EXISTS sources (
+ key TEXT PRIMARY KEY,
+ name TEXT NOT NULL,
+ type TEXT NOT NULL, -- rss | atom | sitemap | jsonld | api | manual
+ category TEXT NOT NULL, -- news | blog | reviews | deals | coupons | trade
+ homepage_url TEXT,
+ endpoints TEXT NOT NULL DEFAULT '[]',
+ auth TEXT NOT NULL DEFAULT '{"mode":"none"}', -- env var NAME only, never a value
+ robots_note TEXT,
+ terms_note TEXT,
+ license_note TEXT,
+ min_interval_ms INTEGER NOT NULL, -- rate limit: gap between two requests to this source
+ poll_interval_sec INTEGER NOT NULL,
+ stale_after_sec INTEGER NOT NULL,
+ max_items INTEGER NOT NULL,
+ enabled INTEGER NOT NULL DEFAULT 0,
+ review_required INTEGER NOT NULL DEFAULT 1,
+ sensitivity TEXT NOT NULL DEFAULT 'none',
+ default_indexability TEXT NOT NULL DEFAULT 'noindex',
+ search_visibility TEXT, -- null = items are not sent to OpenVibe.Search
+ created_at INTEGER NOT NULL,
+ updated_at INTEGER NOT NULL,
+ updated_by TEXT,
+ -- runtime state
+ next_due_at INTEGER NOT NULL DEFAULT 0,
+ not_before INTEGER NOT NULL DEFAULT 0, -- Retry-After / Crawl-delay
+ last_request_at INTEGER,
+ last_run_at INTEGER,
+ last_success_at INTEGER,
+ last_state TEXT,
+ consecutive_failures INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE TABLE IF NOT EXISTS endpoint_state (
+ source_key TEXT NOT NULL,
+ url TEXT NOT NULL,
+ etag TEXT,
+ last_modified TEXT,
+ last_status INTEGER,
+ last_fetch_at INTEGER,
+ PRIMARY KEY (source_key, url)
+);
+
+-- One row per fetch attempt, whatever happened. state is never empty.
+CREATE TABLE IF NOT EXISTS fetch_runs (
+ rid INTEGER PRIMARY KEY AUTOINCREMENT,
+ id TEXT NOT NULL UNIQUE,
+ source_key TEXT NOT NULL,
+ endpoint_url TEXT,
+ trigger TEXT NOT NULL, -- schedule | manual
+ started_at INTEGER NOT NULL,
+ finished_at INTEGER NOT NULL,
+ state TEXT NOT NULL CHECK (state IN ('ok','not_modified','http_error','timeout','robots_denied','parse_error','rate_limited','disabled')),
+ http_status INTEGER,
+ error_code TEXT,
+ detail TEXT,
+ bytes INTEGER,
+ raw_body_hash TEXT,
+ parser_version TEXT,
+ items_seen INTEGER NOT NULL DEFAULT 0,
+ items_created INTEGER NOT NULL DEFAULT 0,
+ items_updated INTEGER NOT NULL DEFAULT 0,
+ items_unchanged INTEGER NOT NULL DEFAULT 0,
+ items_skipped INTEGER NOT NULL DEFAULT 0
+);
+CREATE INDEX IF NOT EXISTS idx_fetch_runs_source ON fetch_runs (source_key, rid);
+
+CREATE TABLE IF NOT EXISTS items (
+ rid INTEGER PRIMARY KEY AUTOINCREMENT,
+ id TEXT NOT NULL UNIQUE,
+ source_key TEXT NOT NULL,
+ category TEXT NOT NULL,
+ identity TEXT NOT NULL, -- guid / @id / url as the source gave it
+ kind TEXT NOT NULL,
+ canonical_url TEXT,
+ title TEXT,
+ summary TEXT,
+ authors TEXT NOT NULL DEFAULT '[]',
+ published_at TEXT,
+ source_updated_at TEXT,
+ fields TEXT NOT NULL DEFAULT '{}',
+ content_hash TEXT NOT NULL,
+ raw_body_hash TEXT,
+ parser_version TEXT NOT NULL,
+ license_note TEXT,
+ terms_note TEXT,
+ first_seen_at INTEGER NOT NULL,
+ retrieved_at INTEGER NOT NULL, -- last successful fetch that contained it
+ revision INTEGER NOT NULL DEFAULT 1,
+ last_fetch_run_id TEXT,
+ entered_by TEXT, -- manual items: the principal that entered it
+ removed_at INTEGER,
+ removed_reason TEXT,
+ change_seq INTEGER NOT NULL,
+ UNIQUE (source_key, identity)
+);
+CREATE INDEX IF NOT EXISTS idx_items_change ON items (change_seq);
+CREATE INDEX IF NOT EXISTS idx_items_source ON items (source_key, change_seq);
+CREATE INDEX IF NOT EXISTS idx_items_category ON items (category, change_seq);
+
+CREATE TABLE IF NOT EXISTS item_revisions (
+ item_id TEXT NOT NULL,
+ revision INTEGER NOT NULL,
+ content_hash TEXT NOT NULL,
+ raw_body_hash TEXT,
+ parser_version TEXT NOT NULL,
+ fetch_run_id TEXT,
+ retrieved_at INTEGER NOT NULL,
+ snapshot TEXT NOT NULL,
+ PRIMARY KEY (item_id, revision)
+);
+
+CREATE TABLE IF NOT EXISTS counters (
+ name TEXT PRIMARY KEY,
+ value INTEGER NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS robots_cache (
+ origin TEXT PRIMARY KEY,
+ fetched_at INTEGER NOT NULL,
+ expires_at INTEGER NOT NULL,
+ status INTEGER,
+ outcome TEXT NOT NULL, -- parsed | unavailable (4xx: allow) | unreachable (deny)
+ rules TEXT NOT NULL DEFAULT '[]',
+ crawl_delay_sec REAL,
+ detail TEXT
+);
+`;
+
+function openDb(dbPath) {
+ if (dbPath !== ':memory:') fs.mkdirSync(path.dirname(path.resolve(dbPath)), { recursive: true });
+ const db = new Database(dbPath);
+ db.pragma('journal_mode = WAL');
+ db.pragma('synchronous = NORMAL');
+ db.pragma('busy_timeout = 5000');
+ db.exec(SCHEMA);
+ outbox.ensureSchema(db);
+ return db;
+}
+
+/** Next value of a named monotonic counter (call inside a transaction). */
+function nextSeq(db, name) {
+ db.prepare('INSERT INTO counters (name, value) VALUES (?, 1) ON CONFLICT(name) DO UPDATE SET value = value + 1').run(name);
+ return db.prepare('SELECT value FROM counters WHERE name = ?').get(name).value;
+}
+
+module.exports = { openDb, nextSeq };
new file mode 100644
new file mode 100644
@@ -0,0 +1,179 @@
@@ -0,0 +1,179 @@
+'use strict';
+/**
+ * Transactional outbox (events.event-envelope@1), OpenVibe.Events client semantics:
+ *
+ * enqueue(envelope) INSIDE the transaction that makes the change, so an event exists exactly
+ * when its effect does (throws outside a transaction)
+ * relay publishes unsent rows to OpenVibe.Events (POST /api/v1/events, service
+ * token for audience openvibe.events with events.event.publish), marks them
+ * sent, backs off on failure; a permanently refused row (4xx other than
+ * 401/408/429) is marked rejected so it cannot block the rest
+ *
+ * The relay runs only when EVENTS_URL is set; without it rows wait and are relayed later. Events
+ * deduplicates on event_id, so a retry after a lost response never publishes twice.
+ */
+const { ids, validate, serviceAuth } = require('openvibe-contracts');
+
+const BACKOFF_MS = [1000, 5000, 30000, 120000, 600000];
+
+function ensureSchema(db) {
+ db.exec(`CREATE TABLE IF NOT EXISTS event_outbox (
+ seq INTEGER PRIMARY KEY AUTOINCREMENT,
+ event_id TEXT NOT NULL UNIQUE,
+ event_type TEXT NOT NULL,
+ envelope TEXT NOT NULL,
+ created_at INTEGER NOT NULL,
+ attempts INTEGER NOT NULL DEFAULT 0,
+ next_attempt_at INTEGER NOT NULL DEFAULT 0,
+ sent_at INTEGER,
+ rejected_at INTEGER,
+ last_error TEXT
+ );
+ CREATE INDEX IF NOT EXISTS idx_event_outbox_due ON event_outbox(sent_at, rejected_at, next_attempt_at);`);
+}
+
+function createOutbox(db, { source, now = () => Date.now() }) {
+ const insert = db.prepare('INSERT INTO event_outbox (event_id, event_type, envelope, created_at) VALUES (?, ?, ?, ?)');
+
+ /** { event_type, subject, payload, visibility?, priority?, actor?, trace_id? } → envelope */
+ function enqueue({ event_type, subject, payload, visibility = 'internal', priority = 'important', actor, trace_id }) {
+ if (!db.inTransaction) throw new Error('outbox.enqueue() must run inside the transaction that makes the change');
+ const ms = now();
+ const env = {
+ event_id: ids.newId('event', ms),
+ event_type,
+ version: 1,
+ source,
+ actor: actor || { type: 'service', id: source },
+ timestamp: new Date(ms).toISOString(),
+ priority,
+ visibility,
+ subject,
+ payload: payload || {},
+ };
+ if (trace_id && /^[0-9a-f]{32}$/.test(trace_id)) env.trace_id = trace_id;
+ const v = validate('events.event-envelope@1', env);
+ if (!v.valid) throw new Error(`outbox: invalid envelope for ${event_type}: ${v.errors.map(e => `${e.path} ${e.message}`).join('; ')}`);
+ insert.run(env.event_id, event_type, JSON.stringify(env), ms);
+ return env;
+ }
+
+ const q = {
+ due: db.prepare('SELECT seq, event_id, envelope, attempts FROM event_outbox WHERE sent_at IS NULL AND rejected_at IS NULL AND next_attempt_at <= ? ORDER BY seq LIMIT ?'),
+ sent: db.prepare('UPDATE event_outbox SET sent_at = ?, attempts = attempts + 1, last_error = NULL WHERE seq = ?'),
+ failed: db.prepare('UPDATE event_outbox SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE seq = ?'),
+ rejected: db.prepare('UPDATE event_outbox SET rejected_at = ?, attempts = attempts + 1, last_error = ? WHERE seq = ?'),
+ pending: db.prepare('SELECT COUNT(*) AS n FROM event_outbox WHERE sent_at IS NULL AND rejected_at IS NULL'),
+ rejectedCount: db.prepare('SELECT COUNT(*) AS n FROM event_outbox WHERE rejected_at IS NOT NULL'),
+ list: db.prepare('SELECT envelope FROM event_outbox ORDER BY seq'),
+ prune: db.prepare('DELETE FROM event_outbox WHERE sent_at IS NOT NULL AND sent_at < ?'),
+ };
+
+ return {
+ enqueue,
+ pending: () => q.pending.get().n,
+ rejected: () => q.rejectedCount.get().n,
+ /** Every envelope in order (tests and operators). */
+ all: () => q.list.all().map(r => JSON.parse(r.envelope)),
+ prune: (olderThanMs = 7 * 24 * 3600 * 1000) => q.prune.run(now() - olderThanMs).changes,
+ _q: q,
+ };
+}
+
+/**
+ * createRelay({ db, outbox, eventsUrl, intervalMs?, tokenClient? | tokenOpts?, fetchImpl?, log?, now? })
+ * → { start, stop, flush, running }
+ * tokenOpts = { tokenUrl, clientId, clientSecret } builds a client-credentials token client for
+ * audience openvibe.events. Without eventsUrl or a token source the relay does nothing.
+ */
+function createRelay({ outbox, eventsUrl, intervalMs = 2000, tokenClient, tokenOpts, fetchImpl = globalThis.fetch, log = console, now = () => Date.now(), db }) {
+ const q = outbox._q;
+ const tokens = tokenClient || (tokenOpts && serviceAuth.createTokenClient({ ...tokenOpts, audience: 'openvibe.events', scope: 'events.event.publish', fetchImpl }));
+ let timer = null;
+ let flushing = null;
+
+ function permanent(status) {
+ return status >= 400 && status < 500 && ![401, 408, 429].includes(status);
+ }
+
+ async function post(rows) {
+ const body = rows.length === 1 ? JSON.parse(rows[0].envelope) : { events: rows.map(r => JSON.parse(r.envelope)) };
+ const res = await fetchImpl(`${eventsUrl}/api/v1/events`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', Accept: 'application/json', ...(await tokens.authHeaders()) },
+ body: JSON.stringify(body),
+ signal: AbortSignal.timeout(10000),
+ });
+ if (res.status === 401 && typeof tokens.invalidate === 'function') tokens.invalidate();
+ const text = await res.text().catch(() => '');
+ return { ok: res.ok, status: res.status, text };
+ }
+
+ function mark(rows, fn) { db.transaction(() => { for (const r of rows) fn(r); })(); }
+
+ async function publishRows(rows) {
+ let r;
+ try {
+ r = await post(rows);
+ } catch (err) {
+ mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], String(err.message).slice(0, 500), row.seq));
+ return { sent: 0, failed: rows.length, rejected: 0 };
+ }
+ if (r.ok) {
+ mark(rows, row => q.sent.run(now(), row.seq));
+ return { sent: rows.length, failed: 0, rejected: 0 };
+ }
+ if (permanent(r.status) && rows.length > 1) {
+ // One bad envelope refuses the batch: isolate it.
+ const total = { sent: 0, failed: 0, rejected: 0 };
+ for (const row of rows) {
+ const s = await publishRows([row]);
+ total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;
+ }
+ return total;
+ }
+ const msg = `${r.status} ${r.text.slice(0, 300)}`;
+ if (permanent(r.status)) {
+ mark(rows, row => q.rejected.run(now(), msg, row.seq));
+ log.warn(`[outbox] event refused by Events: ${msg}`);
+ return { sent: 0, failed: 0, rejected: rows.length };
+ }
+ mark(rows, row => q.failed.run(now() + BACKOFF_MS[Math.min(row.attempts, BACKOFF_MS.length - 1)], msg, row.seq));
+ return { sent: 0, failed: rows.length, rejected: 0 };
+ }
+
+ async function doFlush() {
+ const total = { sent: 0, failed: 0, rejected: 0 };
+ if (!eventsUrl || !tokens) return total;
+ for (;;) {
+ const rows = q.due.all(now(), 50);
+ if (!rows.length) break;
+ const s = await publishRows(rows);
+ total.sent += s.sent; total.failed += s.failed; total.rejected += s.rejected;
+ if (s.failed || rows.length < 50) break;
+ }
+ if (total.failed) log.warn(`[outbox] ${total.failed} event(s) not published yet; will retry`);
+ return total;
+ }
+
+ function flush() {
+ if (!flushing) flushing = doFlush().finally(() => { flushing = null; });
+ return flushing;
+ }
+
+ function start() {
+ if (timer || !eventsUrl) return;
+ timer = setInterval(() => { flush().catch(err => log.warn(`[outbox] relay: ${err.message}`)); }, intervalMs);
+ timer.unref?.();
+ }
+
+ function stop() {
+ if (timer) clearInterval(timer);
+ timer = null;
+ return flushing || Promise.resolve();
+ }
+
+ return { start, stop, flush, running: () => Boolean(timer) };
+}
+
+module.exports = { ensureSchema, createOutbox, createRelay };
new file mode 100644
new file mode 100644
@@ -0,0 +1,94 @@
@@ -0,0 +1,94 @@
+'use strict';
+/**
+ * OpenVibe.Sources entry point.
+ *
+ * node server/index.js (systemd: openvibe-sources.service)
+ *
+ * start() is also what the tests use: it takes a config (server/config.js load()) plus injectable
+ * clock/fetch/log/DNS lookup, and returns handles to every part.
+ */
+const { load } = require('./config');
+const { openDb } = require('./db');
+const { createRegistry } = require('./registry');
+const { createItems } = require('./items');
+const { createGuard } = require('./net/guard');
+const { createFetcher } = require('./net/fetcher');
+const { createSpacer } = require('./spacer');
+const { createRobots } = require('./robots');
+const { createIngest } = require('./ingest');
+const { createScheduler } = require('./scheduler');
+const { createOutbox, createRelay } = require('./events/outbox');
+const { createKeyStore, createAuth } = require('./auth');
+const { createApp } = require('./app');
+
+async function start({ config, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, lookupImpl, log = console, listen = true } = {}) {
+ config = config || load();
+ const db = openDb(config.dbPath);
+ const outbox = createOutbox(db, { source: config.serviceId, now });
+ const relay = createRelay({
+ db, outbox, eventsUrl: config.events.url, intervalMs: config.events.relayIntervalMs, fetchImpl, log, now,
+ tokenClient,
+ tokenOpts: config.oauth.clientSecret ? { tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret } : null,
+ });
+ const registry = createRegistry({ db, now, maxItemsCap: config.items.maxPerFetch });
+ const items = createItems({ db, outbox, now });
+ const guard = createGuard({ allowPrivateHosts: config.fetch.allowPrivateHosts, allowedPorts: config.fetch.allowedPorts, ...(lookupImpl ? { lookupImpl } : {}) });
+ const fetcher = createFetcher({ guard, userAgent: config.fetch.userAgent, timeoutMs: config.fetch.timeoutMs, maxBytes: config.fetch.maxBytes, maxRedirects: config.fetch.maxRedirects });
+ const spacer = createSpacer({ hostMinIntervalMs: config.fetch.hostMinIntervalMs });
+ const robots = createRobots({
+ db, fetcher, agent: config.fetch.robotsAgent, ttlMs: config.fetch.robotsTtlMs, maxBytes: config.fetch.robotsMaxBytes, now,
+ beforeRequest: (host) => spacer.space(host, 0),
+ });
+ const ingest = createIngest({ db, config, registry, items, robots, fetcher, spacer, outbox, now, log, relay });
+ const scheduler = createScheduler({ db, ingest, config, now, log });
+ const keys = createKeyStore({ urls: [config.networkInternalUrl, config.networkUrl], pem: config.networkPublicKey, fetchImpl, log });
+ const auth = createAuth({ config, keys });
+ const app = createApp({ config, db, registry, items, ingest, scheduler, auth, keys, outbox, relay, now, log });
+
+ const keyLoaded = keys.start().catch(() => null);
+ relay.start();
+ if (config.worker.enabled) scheduler.start();
+ const pruneTimer = setInterval(() => { try { outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
+ pruneTimer.unref?.();
+
+ let server = null;
+ if (listen) {
+ server = await new Promise((resolve, reject) => {
+ const s = app.listen(config.port, config.host, () => resolve(s));
+ s.on('error', reject);
+ });
+ log.log(`[sources] listening on http://${config.host}:${server.address().port}`);
+ }
+
+ async function close() {
+ clearInterval(pruneTimer);
+ keys.stop();
+ await scheduler.stop();
+ await relay.stop();
+ if (server) {
+ server.closeAllConnections?.();
+ await new Promise(resolve => server.close(() => resolve()));
+ }
+ db.close();
+ }
+
+ return { config, db, registry, items, guard, fetcher, robots, ingest, scheduler, outbox, relay, keys, keyLoaded, auth, app, server, close };
+}
+
+if (require.main === module) {
+ require('dotenv').config();
+ start().then((handles) => {
+ const shutdown = (sig) => {
+ console.log(`[sources] ${sig}: shutting down`);
+ handles.close().then(() => process.exit(0), () => process.exit(1));
+ setTimeout(() => process.exit(1), 20000).unref();
+ };
+ process.on('SIGTERM', () => shutdown('SIGTERM'));
+ process.on('SIGINT', () => shutdown('SIGINT'));
+ }).catch((err) => {
+ console.error(`[sources] failed to start: ${err.stack || err}`);
+ process.exit(1);
+ });
+}
+
+module.exports = { start };
new file mode 100644
new file mode 100644
@@ -0,0 +1,268 @@
@@ -0,0 +1,268 @@
+'use strict';
+/**
+ * Ingestion runs: one source at a time (never two runs of the same source in flight), each
+ * endpoint fetched in order, every attempt recorded as a fetch_runs row with an explicit state:
+ *
+ * ok fetched and parsed; items created/updated/unchanged
+ * not_modified 304 to a conditional GET (ETag / Last-Modified from the last good parse)
+ * http_error any other status, or a refused/failed connection, oversize body, bad redirect
+ * timeout no complete response within SOURCES_FETCH_TIMEOUT_MS
+ * robots_denied robots.txt disallows the URL (or a redirect target), or is unreachable
+ * parse_error the body is not what the adapter reads
+ * rate_limited the source's rate limit or a Retry-After has not elapsed (nothing requested),
+ * or the site answered 429
+ * disabled the source is disabled, or its credential variable is not set
+ *
+ * A failed run never creates or modifies an item: only the `ok` path calls items.ingest(), in the
+ * same transaction as its run row. Validators (ETag/Last-Modified) are stored only after a good
+ * parse, so a body that failed to parse is fetched and parsed again next time, not skipped as 304.
+ */
+const { ids } = require('openvibe-contracts');
+const { ADAPTERS } = require('./adapters');
+const { sha256, decodeBody } = require('./adapters/util');
+
+const FAILURE_STATES = new Set(['http_error', 'timeout', 'robots_denied', 'parse_error', 'rate_limited']);
+
+function retryAfterMs(value, now) {
+ if (value == null) return null;
+ const s = String(value).trim();
+ if (/^\d{1,7}$/.test(s)) return Number(s) * 1000;
+ const t = Date.parse(s);
+ return Number.isFinite(t) ? Math.max(0, t - now) : null;
+}
+
+function createIngest({ db, config, registry, items, robots, fetcher, spacer, outbox, now = () => Date.now(), log = console, relay = null }) {
+ const inflight = new Set();
+ const { space, setCrawlDelay } = spacer;
+
+ const st = {
+ insertRun: db.prepare(`INSERT INTO fetch_runs (id, source_key, endpoint_url, trigger, started_at, finished_at, state, http_status,
+ error_code, detail, bytes, raw_body_hash, parser_version, items_seen, items_created, items_updated, items_unchanged, items_skipped)
+ VALUES (@id, @source_key, @endpoint_url, @trigger, @started_at, @finished_at, @state, @http_status, @error_code, @detail, @bytes,
+ @raw_body_hash, @parser_version, @items_seen, @items_created, @items_updated, @items_unchanged, @items_skipped)`),
+ endpoint: db.prepare('SELECT * FROM endpoint_state WHERE source_key = ? AND url = ?'),
+ saveValidators: db.prepare(`INSERT INTO endpoint_state (source_key, url, etag, last_modified, last_status, last_fetch_at) VALUES (@k, @u, @etag, @lm, @status, @at)
+ ON CONFLICT(source_key, url) DO UPDATE SET etag = excluded.etag, last_modified = excluded.last_modified, last_status = excluded.last_status, last_fetch_at = excluded.last_fetch_at`),
+ touchEndpoint: db.prepare(`INSERT INTO endpoint_state (source_key, url, last_status, last_fetch_at) VALUES (@k, @u, @status, @at)
+ ON CONFLICT(source_key, url) DO UPDATE SET last_status = excluded.last_status, last_fetch_at = excluded.last_fetch_at`),
+ requested: db.prepare('UPDATE sources SET last_request_at = ? WHERE key = ?'),
+ finish: db.prepare(`UPDATE sources SET last_run_at = @at, last_state = @state, consecutive_failures = @failures,
+ last_success_at = COALESCE(@success_at, last_success_at), next_due_at = @next_due, not_before = @not_before WHERE key = @key`),
+ };
+
+ // ── Run rows ─────────────────────────────────────────────
+
+ function baseRun(source, endpointUrl, trigger, startedAt) {
+ return {
+ id: `frn_${ids.ulid(startedAt)}`, source_key: source.key, endpoint_url: endpointUrl, trigger,
+ started_at: startedAt, finished_at: now(), state: null, http_status: null, error_code: null, detail: null,
+ bytes: null, raw_body_hash: null, parser_version: null,
+ items_seen: 0, items_created: 0, items_updated: 0, items_unchanged: 0, items_skipped: 0,
+ };
+ }
+
+ function failedEvent(source, run, failures) {
+ outbox.enqueue({
+ event_type: 'sources.fetch.failed',
+ subject: { type: 'source', id: source.key },
+ payload: {
+ source_key: source.key, category: source.category, run_id: run.id, endpoint_url: run.endpoint_url,
+ state: run.state, error_code: run.error_code, http_status: run.http_status, consecutive_failures: failures,
+ },
+ });
+ }
+
+ /** Record a run that changed no item (every non-ok outcome, and 304). */
+ function recordNoItems(source, run, { countsAsFailure, failures }) {
+ db.transaction(() => {
+ st.insertRun.run(run);
+ if (run.endpoint_url && run.http_status != null) st.touchEndpoint.run({ k: source.key, u: run.endpoint_url, status: run.http_status, at: run.finished_at });
+ if (countsAsFailure) failedEvent(source, run, failures);
+ })();
+ return run;
+ }
+
+ // ── One endpoint ─────────────────────────────────────────
+
+ async function fetchEndpoint(source, endpoint, trigger) {
+ const adapter = ADAPTERS[source.type];
+ const startedAt = now();
+ const run = baseRun(source, endpoint.url, trigger, startedAt);
+ const target = new URL(endpoint.url);
+ const host = target.host;
+
+ // credential by variable name, read now, never stored
+ const headers = { Accept: adapter.accept };
+ const sensitive = [];
+ let requestUrl = target;
+ if (source.auth.mode !== 'none') {
+ const secret = config.secrets[source.auth.env];
+ if (!secret) {
+ Object.assign(run, { state: 'disabled', error_code: 'credential_missing', detail: `environment variable ${source.auth.env} is not set`, finished_at: now() });
+ return { run, failed: false };
+ }
+ if (source.auth.mode === 'bearer') { headers.Authorization = `Bearer ${secret}`; sensitive.push('Authorization'); }
+ if (source.auth.mode === 'header') { headers[source.auth.header] = secret; sensitive.push(source.auth.header); }
+ if (source.auth.mode === 'query') { requestUrl = new URL(target); requestUrl.searchParams.set(source.auth.param, secret); }
+ }
+
+ // robots.txt for the endpoint's origin (a robots fetch is a request to the host too)
+ const verdict = await robots.check(target.toString());
+ setCrawlDelay(host, verdict.crawlDelaySec);
+ if (!verdict.allowed) {
+ if (verdict.refusedCode) Object.assign(run, { state: 'http_error', error_code: verdict.refusedCode, detail: `${host} is not fetchable (${verdict.refusedCode})`, finished_at: now() });
+ else Object.assign(run, { state: 'robots_denied', error_code: 'robots', detail: verdict.reason, finished_at: now() });
+ return { run, failed: true };
+ }
+
+ const validators = st.endpoint.get(source.key, endpoint.url) || {};
+ await space(host, source.min_interval_ms);
+ st.requested.run(now(), source.key);
+ let res;
+ try {
+ res = await fetcher.fetchUrl(requestUrl.toString(), {
+ headers,
+ etag: validators.etag || null,
+ lastModified: validators.last_modified || null,
+ sensitiveHeaders: sensitive,
+ // every redirect hop is checked against its own origin's robots.txt and, on a new
+ // host, spaced like any other request to that host
+ beforeHop: async (next) => {
+ const v = await robots.check(next.toString());
+ if (!v.allowed) return { refuse: v.reason };
+ if (next.host !== host) {
+ setCrawlDelay(next.host, v.crawlDelaySec);
+ await space(next.host, source.min_interval_ms);
+ }
+ return null;
+ },
+ });
+ } catch (err) {
+ const state = err.code === 'timeout' ? 'timeout' : err.code === 'hop_refused' ? 'robots_denied' : 'http_error';
+ Object.assign(run, { state, error_code: err.code || 'network', detail: String(err.message).slice(0, 500), finished_at: now() });
+ return { run, failed: true };
+ }
+ run.http_status = res.status;
+ run.bytes = res.bytes != null ? res.bytes : null;
+ run.finished_at = now();
+
+ if (res.status === 304) {
+ Object.assign(run, { state: 'not_modified' });
+ return { run, failed: false, success: true };
+ }
+ if (res.status === 429) {
+ const wait = Math.min(retryAfterMs(res.headers['retry-after'], now()) ?? 10 * 60 * 1000, 24 * 3600 * 1000);
+ Object.assign(run, { state: 'rate_limited', error_code: 'upstream_429', detail: `the site asked us to wait ${Math.round(wait / 1000)} s` });
+ return { run, failed: true, notBefore: now() + wait };
+ }
+ if (res.status !== 200 && res.status !== 203) {
+ const wait = res.status === 503 ? retryAfterMs(res.headers['retry-after'], now()) : null;
+ Object.assign(run, { state: 'http_error', error_code: `http_${res.status}`, detail: `HTTP ${res.status}` });
+ return { run, failed: true, notBefore: wait != null ? now() + Math.min(wait, 24 * 3600 * 1000) : null };
+ }
+
+ run.raw_body_hash = sha256(res.body);
+ run.parser_version = adapter.version;
+ let parsed;
+ try {
+ const text = decodeBody(res.body, res.headers['content-type']);
+ parsed = adapter.parse(text, { url: res.finalUrl, endpoint, summaryMax: config.items.summaryMax, maxItems: source.max_items });
+ } catch (err) {
+ Object.assign(run, { state: 'parse_error', error_code: err.code === 'parse_error' ? 'unreadable' : 'adapter_error', detail: String(err.message).slice(0, 500) });
+ if (err.code !== 'parse_error') log.error(`[ingest] ${source.key}: adapter crashed: ${err.stack || err}`);
+ return { run, failed: true };
+ }
+
+ // The only path that touches items: the run row, items, validators and events commit together.
+ db.transaction(() => {
+ const counts = items.ingest(source, parsed.items, {
+ runId: run.id, rawBodyHash: run.raw_body_hash, parserVersion: adapter.version, retrievedAt: run.finished_at,
+ });
+ Object.assign(run, {
+ state: 'ok', items_seen: parsed.items.length, items_created: counts.created, items_updated: counts.updated,
+ items_unchanged: counts.unchanged, items_skipped: parsed.skipped + counts.skipped,
+ });
+ st.insertRun.run(run);
+ st.saveValidators.run({
+ k: source.key, u: endpoint.url, etag: res.headers.etag || null, lm: res.headers['last-modified'] || null,
+ status: res.status, at: run.finished_at,
+ });
+ })();
+ return { run, failed: false, success: true, recorded: true };
+ }
+
+ // ── One source ───────────────────────────────────────────
+
+ /**
+ * run(key, { trigger }) → { runs: [...] } | { busy: true } | { manual: true } | null (unknown)
+ */
+ async function run(key, { trigger = 'schedule' } = {}) {
+ const row = registry.get(key);
+ if (!row) return null;
+ if (row.type === 'manual') return { manual: true };
+ if (inflight.has(key)) return { busy: true };
+ inflight.add(key);
+ try {
+ const source = registry.fromRow(row);
+ const t = now();
+ const runs = [];
+
+ if (!source.enabled) {
+ const r = baseRun(source, null, trigger, t);
+ Object.assign(r, { state: 'disabled', error_code: 'source_disabled', detail: 'the source is disabled' });
+ runs.push(recordNoItems(source, r, { countsAsFailure: false }));
+ return { runs };
+ }
+ const sinceLast = row.last_request_at == null ? Infinity : t - row.last_request_at;
+ if (sinceLast < source.min_interval_ms || t < row.not_before) {
+ const r = baseRun(source, null, trigger, t);
+ const until = Math.max(row.not_before, (row.last_request_at || 0) + source.min_interval_ms);
+ Object.assign(r, { state: 'rate_limited', error_code: 'local_rate_limit', detail: `next request allowed at ${new Date(until).toISOString()}` });
+ runs.push(recordNoItems(source, r, { countsAsFailure: false }));
+ return { runs, rateLimitedUntil: until };
+ }
+
+ let anyFailure = false;
+ let anySuccess = false;
+ let notBefore = 0;
+ let failures = row.consecutive_failures;
+ for (const endpoint of source.endpoints) {
+ let out;
+ try {
+ out = await fetchEndpoint(source, endpoint, trigger);
+ } catch (err) {
+ // Anything unexpected is still a recorded failure, never a silent gap.
+ const r = baseRun(source, endpoint.url, trigger, t);
+ Object.assign(r, { state: 'http_error', error_code: 'internal', detail: String(err.message).slice(0, 300), finished_at: now() });
+ log.error(`[ingest] ${key}: ${err.stack || err}`);
+ out = { run: r, failed: true };
+ }
+ if (out.failed) { anyFailure = true; failures = row.consecutive_failures + 1; }
+ if (out.success) anySuccess = true;
+ if (out.notBefore) notBefore = Math.max(notBefore, out.notBefore);
+ if (!out.recorded) recordNoItems(source, out.run, { countsAsFailure: out.failed, failures });
+ runs.push(out.run);
+ if (out.run.state === 'rate_limited') break; // the site asked us to stop
+ }
+
+ const finishedAt = now();
+ const newFailures = anyFailure ? row.consecutive_failures + 1 : 0;
+ const backoff = anyFailure
+ ? Math.min(source.poll_interval_sec * 1000 * 2 ** Math.min(newFailures - 1, 10), Math.max(config.worker.maxBackoffMs, source.poll_interval_sec * 1000))
+ : source.poll_interval_sec * 1000;
+ const lastState = runs.map(r => r.state).find(s => FAILURE_STATES.has(s)) || runs[runs.length - 1].state;
+ st.finish.run({
+ key, at: finishedAt, state: lastState, failures: newFailures,
+ success_at: anySuccess ? finishedAt : null, next_due: finishedAt + backoff, not_before: notBefore,
+ });
+ if (relay) relay.flush().catch(() => {});
+ return { runs };
+ } finally {
+ inflight.delete(key);
+ }
+ }
+
+ return { run, inflight: () => [...inflight] };
+}
+
+module.exports = { createIngest, FAILURE_STATES, retryAfterMs };
new file mode 100644
new file mode 100644
@@ -0,0 +1,239 @@
@@ -0,0 +1,239 @@
+'use strict';
+/**
+ * Source items: provenance-carrying records of what a source said, when we retrieved it, and
+ * from which exact body and parser.
+ *
+ * Every item stores its source key, canonical URL, retrieval time (first seen, last seen), a
+ * content hash of the parsed fields, the hash of the raw body it was parsed from, the parser
+ * version and the source's license/terms notes as they were at ingestion. A changed content hash
+ * is a new revision (the previous one stays in item_revisions); an unchanged item only has its
+ * retrieval time refreshed. Removal (a takedown, a licence problem) is explicit, audited and
+ * sticky: a later fetch that still lists the item does not bring it back.
+ *
+ * Only successful fetches reach ingest(); nothing in this module runs for a failed fetch.
+ */
+const { ids } = require('openvibe-contracts');
+const { nextSeq } = require('./db');
+const { sha256, canonicalJson } = require('./adapters/util');
+
+const STAFF_GROUPS = ['role:admin', 'role:global_mod'];
+
+function contentHash(it) {
+ return sha256(canonicalJson({
+ kind: it.kind, canonical_url: it.canonical_url || null, title: it.title || null, summary: it.summary || null,
+ authors: it.authors || [], published_at: it.published_at || null, updated_at: it.updated_at || null, fields: it.fields || {},
+ }));
+}
+
+function createItems({ db, outbox, now = () => Date.now() }) {
+ const st = {
+ byIdentity: db.prepare('SELECT * FROM items WHERE source_key = ? AND identity = ?'),
+ byId: db.prepare('SELECT * FROM items WHERE id = ?'),
+ insert: db.prepare(`INSERT INTO items (id, source_key, category, identity, kind, canonical_url, title, summary, authors, published_at,
+ source_updated_at, fields, content_hash, raw_body_hash, parser_version, license_note, terms_note, first_seen_at, retrieved_at,
+ revision, last_fetch_run_id, entered_by, change_seq)
+ VALUES (@id, @source_key, @category, @identity, @kind, @canonical_url, @title, @summary, @authors, @published_at,
+ @source_updated_at, @fields, @content_hash, @raw_body_hash, @parser_version, @license_note, @terms_note, @first_seen_at, @retrieved_at,
+ 1, @last_fetch_run_id, @entered_by, @change_seq)`),
+ update: db.prepare(`UPDATE items SET kind = @kind, canonical_url = @canonical_url, title = @title, summary = @summary, authors = @authors,
+ published_at = @published_at, source_updated_at = @source_updated_at, fields = @fields, content_hash = @content_hash,
+ raw_body_hash = @raw_body_hash, parser_version = @parser_version, license_note = @license_note, terms_note = @terms_note,
+ retrieved_at = @retrieved_at, revision = revision + 1, last_fetch_run_id = @last_fetch_run_id, change_seq = @change_seq
+ WHERE id = @id`),
+ seen: db.prepare('UPDATE items SET retrieved_at = ?, last_fetch_run_id = ? WHERE id = ?'),
+ remove: db.prepare('UPDATE items SET removed_at = ?, removed_reason = ?, revision = revision + 1, change_seq = ? WHERE id = ?'),
+ revision: db.prepare(`INSERT INTO item_revisions (item_id, revision, content_hash, raw_body_hash, parser_version, fetch_run_id, retrieved_at, snapshot)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?)`),
+ revisions: db.prepare('SELECT * FROM item_revisions WHERE item_id = ? ORDER BY revision'),
+ };
+
+ function view(row) {
+ if (!row) return null;
+ const iso = (v) => (v == null ? null : new Date(v).toISOString());
+ return {
+ id: row.id,
+ source_key: row.source_key,
+ category: row.category,
+ kind: row.kind,
+ identity: row.identity,
+ canonical_url: row.canonical_url,
+ title: row.title,
+ summary: row.summary,
+ authors: JSON.parse(row.authors),
+ published_at: row.published_at,
+ source_updated_at: row.source_updated_at,
+ fields: JSON.parse(row.fields),
+ revision: row.revision,
+ provenance: {
+ retrieved_at: iso(row.retrieved_at),
+ first_seen_at: iso(row.first_seen_at),
+ content_hash: row.content_hash,
+ raw_body_hash: row.raw_body_hash,
+ parser_version: row.parser_version,
+ fetch_run_id: row.last_fetch_run_id,
+ license_note: row.license_note,
+ terms_note: row.terms_note,
+ entered_by: row.entered_by,
+ },
+ removed: row.removed_at ? { at: iso(row.removed_at), reason: row.removed_reason } : null,
+ change_seq: row.change_seq,
+ };
+ }
+
+ function subjectOf(row) {
+ return { type: 'item', id: row.id, revision: row.revision };
+ }
+
+ function itemEvent(type, row, extra = {}) {
+ outbox.enqueue({
+ event_type: type,
+ subject: subjectOf(row),
+ payload: {
+ item_id: row.id, source_key: row.source_key, category: row.category, kind: row.kind,
+ canonical_url: row.canonical_url, title: row.title, revision: row.revision, content_hash: row.content_hash,
+ retrieved_at: new Date(row.retrieved_at).toISOString(), ...extra,
+ },
+ });
+ }
+
+ /** search.index-document@1 for a raw item: staff-only, never indexable, provenance attached. */
+ function indexDocument(row) {
+ const title = row.title || row.canonical_url || row.identity;
+ const doc = {
+ owner: 'sources', type: 'item', id: row.id, revision: row.revision,
+ visibility: 'members', acl: { groups: STAFF_GROUPS },
+ title: String(title).slice(0, 500),
+ summary: row.summary ? row.summary.slice(0, 4000) : '',
+ body: '',
+ facets: { category: row.category, source: row.source_key, kind: row.kind },
+ authorship: 'imported',
+ provenance: [{
+ service: 'sources', type: 'item', id: row.id, revision: row.revision,
+ ...(row.canonical_url ? { url: row.canonical_url } : {}),
+ retrieved_at: new Date(row.retrieved_at).toISOString(),
+ }],
+ publication_state: 'published',
+ published_at: row.published_at || null,
+ updated_at: row.source_updated_at || null,
+ indexability: { decision: 'noindex', reasons: ['third_party_content'] },
+ };
+ if (row.canonical_url) doc.canonical_url = row.canonical_url;
+ return doc;
+ }
+
+ function indexEvent(source, row, deleted = false) {
+ if (!source.search_visibility) return;
+ outbox.enqueue({
+ event_type: deleted ? 'sources.index_document.deleted' : 'sources.index_document.upserted',
+ subject: subjectOf(row),
+ payload: deleted ? { type: 'item', id: row.id, revision: row.revision } : indexDocument(row),
+ });
+ }
+
+ function snapshot(row) {
+ return JSON.stringify({
+ kind: row.kind, canonical_url: row.canonical_url, title: row.title, summary: row.summary,
+ authors: JSON.parse(row.authors), published_at: row.published_at, source_updated_at: row.source_updated_at, fields: JSON.parse(row.fields),
+ });
+ }
+
+ function values(source, it, ctx) {
+ return {
+ kind: it.kind,
+ canonical_url: it.canonical_url || null,
+ title: it.title || null,
+ summary: it.summary || null,
+ authors: JSON.stringify(it.authors || []),
+ published_at: it.published_at || null,
+ source_updated_at: it.updated_at || null,
+ fields: JSON.stringify(it.fields || {}),
+ content_hash: contentHash(it),
+ raw_body_hash: ctx.rawBodyHash || null,
+ parser_version: ctx.parserVersion,
+ license_note: source.license_note || null,
+ terms_note: source.terms_note || null,
+ retrieved_at: ctx.retrievedAt,
+ last_fetch_run_id: ctx.runId || null,
+ };
+ }
+
+ /**
+ * Apply one successful fetch's parsed items. Must run inside the transaction that also writes
+ * the fetch run row. → { created, updated, unchanged, skipped }
+ */
+ function ingest(source, parsedItems, ctx) {
+ if (!db.inTransaction) throw new Error('items.ingest() must run inside the fetch run transaction');
+ const counts = { created: 0, updated: 0, unchanged: 0, skipped: 0 };
+ for (const it of parsedItems) {
+ const cur = st.byIdentity.get(source.key, it.identity);
+ const v = values(source, it, ctx);
+ if (!cur) {
+ const row = { id: `itm_${ids.ulid(ctx.retrievedAt)}`, source_key: source.key, category: source.category, identity: it.identity, first_seen_at: ctx.retrievedAt, entered_by: ctx.enteredBy || null, change_seq: nextSeq(db, 'items'), ...v };
+ st.insert.run(row);
+ const saved = st.byId.get(row.id);
+ st.revision.run(saved.id, 1, saved.content_hash, saved.raw_body_hash, saved.parser_version, ctx.runId || null, ctx.retrievedAt, snapshot(saved));
+ itemEvent('sources.item.created', saved);
+ indexEvent(source, saved);
+ counts.created++;
+ } else if (cur.removed_at) {
+ counts.skipped++; // removal is sticky
+ } else if (cur.content_hash !== v.content_hash) {
+ st.update.run({ ...v, id: cur.id, change_seq: nextSeq(db, 'items') });
+ const saved = st.byId.get(cur.id);
+ st.revision.run(saved.id, saved.revision, saved.content_hash, saved.raw_body_hash, saved.parser_version, ctx.runId || null, ctx.retrievedAt, snapshot(saved));
+ itemEvent('sources.item.updated', saved, { previous_content_hash: cur.content_hash });
+ indexEvent(source, saved);
+ counts.updated++;
+ } else {
+ st.seen.run(ctx.retrievedAt, ctx.runId || null, cur.id);
+ counts.unchanged++;
+ }
+ }
+ return counts;
+ }
+
+ /** Remove an item (takedown, licence, error). Sticky; emits sources.item.removed. */
+ function remove(source, id, reason, by) {
+ return db.transaction(() => {
+ const cur = st.byId.get(id);
+ if (!cur || cur.source_key !== source.key) return null;
+ if (cur.removed_at) return view(cur);
+ const t = now();
+ st.remove.run(t, `${reason}${by ? ` (by ${by})` : ''}`.slice(0, 500), nextSeq(db, 'items'), id);
+ const saved = st.byId.get(id);
+ st.revision.run(saved.id, saved.revision, saved.content_hash, saved.raw_body_hash, saved.parser_version, null, t, JSON.stringify({ removed: true, reason }));
+ itemEvent('sources.item.removed', saved, { reason });
+ indexEvent(source, saved, true);
+ return view(saved);
+ })();
+ }
+
+ function get(id, { revisions = false } = {}) {
+ const row = st.byId.get(id);
+ if (!row) return null;
+ const out = view(row);
+ if (revisions) {
+ out.revisions = st.revisions.all(id).map(r => ({
+ revision: r.revision, content_hash: r.content_hash, raw_body_hash: r.raw_body_hash, parser_version: r.parser_version,
+ fetch_run_id: r.fetch_run_id, retrieved_at: new Date(r.retrieved_at).toISOString(), snapshot: JSON.parse(r.snapshot),
+ }));
+ }
+ return out;
+ }
+
+ /** Items in change order (created, updated or removed after `after`). */
+ function list({ source = null, category = null, after = 0, limit = 100, includeRemoved = false } = {}) {
+ const rows = db.prepare(`SELECT * FROM items WHERE change_seq > @after
+ AND (@source IS NULL OR source_key = @source) AND (@category IS NULL OR category = @category)
+ AND (@incl = 1 OR removed_at IS NULL)
+ ORDER BY change_seq LIMIT @limit`).all({ after, source, category, incl: includeRemoved ? 1 : 0, limit: limit + 1 });
+ const more = rows.length > limit;
+ const page = rows.slice(0, limit);
+ // next_after is where to resume, also when this page is the last one (poll with it later).
+ return { items: page.map(view), next_after: page.length ? page[page.length - 1].change_seq : after, more };
+ }
+
+ return { ingest, remove, get, list, view, indexDocument, contentHash };
+}
+
+module.exports = { createItems, contentHash, STAFF_GROUPS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,116 @@
@@ -0,0 +1,116 @@
+'use strict';
+/**
+ * The only way Sources reaches the internet: GET with an address guard (server/net/guard.js),
+ * manual redirects re-checked hop by hop, conditional headers, one overall deadline covering
+ * connect + headers + body, a byte cap applied to the decoded body, and no cookies.
+ *
+ * fetchUrl(url, { headers, etag, lastModified, beforeHop(url) }) →
+ * { kind: 'response', status, headers, body (Buffer|null), finalUrl, bytes }
+ * or throws FetchError { code: timeout | too_large | address_refused | port_refused | bad_url |
+ * too_many_redirects | network | hop_refused, detail }
+ */
+const http = require('http');
+const https = require('https');
+const zlib = require('zlib');
+const { GuardError } = require('./guard');
+
+class FetchError extends Error {
+ constructor(code, detail) { super(detail); this.code = code; }
+}
+
+function createFetcher({ guard, userAgent, timeoutMs = 20000, maxBytes = 5 * 1024 * 1024, maxRedirects = 5 }) {
+ function once(url, headers, deadline, capBytes) {
+ return new Promise((resolve, reject) => {
+ const mod = url.protocol === 'https:' ? https : http;
+ const remaining = deadline - Date.now();
+ if (remaining <= 0) return reject(new FetchError('timeout', 'deadline passed'));
+ let settled = false;
+ const done = (fn, v) => { if (!settled) { settled = true; clearTimeout(timer); fn(v); } };
+ const req = mod.request(url, {
+ method: 'GET',
+ headers,
+ lookup: guard.lookup,
+ agent: false,
+ timeout: remaining,
+ }, (res) => {
+ const status = res.statusCode;
+ if (status >= 300 && status < 400 && res.headers.location) {
+ res.resume();
+ return done(resolve, { redirect: res.headers.location, status, headers: res.headers });
+ }
+ const declared = Number(res.headers['content-length']);
+ if (Number.isFinite(declared) && declared > capBytes && status === 200) {
+ req.destroy();
+ return done(reject, new FetchError('too_large', `content-length ${declared} exceeds ${capBytes} bytes`));
+ }
+ let stream = res;
+ const enc = String(res.headers['content-encoding'] || '').trim().toLowerCase();
+ if (enc === 'gzip' || enc === 'x-gzip') stream = res.pipe(zlib.createGunzip());
+ else if (enc === 'deflate') stream = res.pipe(zlib.createInflate());
+ else if (enc === 'br') stream = res.pipe(zlib.createBrotliDecompress());
+ const chunks = [];
+ let bytes = 0;
+ stream.on('data', (c) => {
+ bytes += c.length;
+ if (bytes > capBytes) {
+ req.destroy();
+ done(reject, new FetchError('too_large', `body exceeds ${capBytes} bytes`));
+ return;
+ }
+ chunks.push(c);
+ });
+ stream.on('end', () => done(resolve, { status, headers: res.headers, body: Buffer.concat(chunks), bytes }));
+ stream.on('error', (err) => done(reject, new FetchError('network', `body: ${err.message}`)));
+ res.on('aborted', () => done(reject, new FetchError('network', 'response aborted')));
+ });
+ const timer = setTimeout(() => {
+ req.destroy();
+ done(reject, new FetchError('timeout', `no complete response within ${timeoutMs} ms`));
+ }, remaining);
+ req.on('timeout', () => {
+ req.destroy();
+ done(reject, new FetchError('timeout', `no complete response within ${timeoutMs} ms`));
+ });
+ req.on('error', (err) => {
+ if (err instanceof GuardError) return done(reject, new FetchError(err.code, err.message));
+ done(reject, new FetchError('network', `${err.code || 'error'}: ${err.message}`));
+ });
+ req.end();
+ });
+ }
+
+ async function fetchUrl(input, { headers = {}, etag = null, lastModified = null, beforeHop = null, capBytes = maxBytes, sensitiveHeaders = [] } = {}) {
+ const deadline = Date.now() + timeoutMs;
+ let url;
+ try { url = guard.checkUrl(input); } catch (err) { throw new FetchError(err.code || 'bad_url', err.message); }
+ const h = {
+ 'User-Agent': userAgent,
+ Accept: headers.Accept || '*/*',
+ 'Accept-Encoding': 'gzip, deflate, br',
+ ...headers,
+ };
+ if (etag) h['If-None-Match'] = etag;
+ if (lastModified) h['If-Modified-Since'] = lastModified;
+ for (let hop = 0; ; hop++) {
+ const r = await once(url, h, deadline, capBytes);
+ if (!r.redirect) return { kind: 'response', status: r.status, headers: r.headers, body: r.body, bytes: r.bytes, finalUrl: url.toString() };
+ if (hop >= maxRedirects) throw new FetchError('too_many_redirects', `more than ${maxRedirects} redirects`);
+ let next;
+ try { next = guard.checkUrl(new URL(r.redirect, url).toString()); } catch (err) { throw new FetchError(err.code || 'bad_url', `redirect: ${err.message}`); }
+ if (beforeHop) {
+ const verdict = await beforeHop(next);
+ if (verdict && verdict.refuse) throw new FetchError('hop_refused', verdict.refuse);
+ }
+ // Conditional headers belong to the original resource only; credentials never leave
+ // the origin they were configured for.
+ delete h['If-None-Match'];
+ delete h['If-Modified-Since'];
+ if (next.origin !== url.origin) for (const name of sensitiveHeaders) delete h[name];
+ url = next;
+ }
+ }
+
+ return { fetchUrl };
+}
+
+module.exports = { createFetcher, FetchError };
new file mode 100644
new file mode 100644
@@ -0,0 +1,114 @@
@@ -0,0 +1,114 @@
+'use strict';
+/**
+ * Outbound address policy (roadmap anti-goal 13: no fetcher may reach internal addresses).
+ *
+ * A URL is fetchable only when it is http(s), on an allowed port, and every address its host
+ * resolves to is public unicast. The check runs at connect time through the socket's `lookup`
+ * (so a DNS answer cannot change between check and connect) and up front for IP literals (which
+ * never go through lookup). SOURCES_ALLOW_PRIVATE_HOSTS lists exact hostnames/IPs exempted —
+ * for tests and deliberate on-host sources only.
+ */
+const dns = require('dns');
+const net = require('net');
+
+const blocked = new net.BlockList();
+for (const [addr, prefix] of [
+ ['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8], ['169.254.0.0', 16],
+ ['172.16.0.0', 12], ['192.0.0.0', 24], ['192.0.2.0', 24], ['192.88.99.0', 24], ['192.168.0.0', 16],
+ ['198.18.0.0', 15], ['198.51.100.0', 24], ['203.0.113.0', 24], ['224.0.0.0', 4], ['240.0.0.0', 4],
+]) blocked.addSubnet(addr, prefix, 'ipv4');
+for (const [addr, prefix] of [
+ ['::', 128], ['::1', 128], ['100::', 64], ['2001:db8::', 32], ['2001::', 32], ['fc00::', 7],
+ ['fe80::', 10], ['fec0::', 10], ['ff00::', 8],
+]) blocked.addSubnet(addr, prefix, 'ipv6');
+
+class GuardError extends Error {
+ constructor(code, detail) { super(detail); this.code = code; }
+}
+
+/** IPv4 embedded in IPv4-mapped (::ffff:a.b.c.d), NAT64 (64:ff9b::/96) or 6to4 (2002::/16). */
+function embeddedV4(ip) {
+ const lower = ip.toLowerCase();
+ const mapped = /^::ffff:(\d+\.\d+\.\d+\.\d+)$/.exec(lower);
+ if (mapped) return mapped[1];
+ const full = expandV6(lower);
+ if (!full) return null;
+ const hex = (i) => parseInt(full[i], 16);
+ const v4 = (hi, lo) => `${hi >> 8}.${hi & 255}.${lo >> 8}.${lo & 255}`;
+ if (full.slice(0, 5).every(h => h === '0000') && full[5] === 'ffff') return v4(hex(6), hex(7));
+ if (full[0] === '0064' && full[1] === 'ff9b' && full.slice(2, 6).every(h => h === '0000')) return v4(hex(6), hex(7));
+ if (full[0] === '2002') return v4(hex(1), hex(2));
+ return null;
+}
+
+function expandV6(ip) {
+ if (!net.isIPv6(ip)) return null;
+ let s = ip;
+ const v4tail = /(\d+\.\d+\.\d+\.\d+)$/.exec(s);
+ if (v4tail) {
+ const p = v4tail[1].split('.').map(Number);
+ s = s.slice(0, -v4tail[1].length) + ((p[0] << 8) | p[1]).toString(16) + ':' + ((p[2] << 8) | p[3]).toString(16);
+ }
+ const [head, tail] = s.split('::');
+ const h = head ? head.split(':') : [];
+ const t = tail !== undefined ? (tail ? tail.split(':') : []) : [];
+ const fill = tail !== undefined ? Array(8 - h.length - t.length).fill('0') : [];
+ const parts = [...h, ...fill, ...t];
+ if (parts.length !== 8) return null;
+ return parts.map(x => x.padStart(4, '0'));
+}
+
+/** Is this IP address public unicast? */
+function isPublicAddress(ip) {
+ const family = net.isIP(ip);
+ if (family === 4) return !blocked.check(ip, 'ipv4');
+ if (family === 6) {
+ const v4 = embeddedV4(ip);
+ if (v4) return !blocked.check(v4, 'ipv4');
+ return !blocked.check(ip, 'ipv6');
+ }
+ return false;
+}
+
+function createGuard({ allowPrivateHosts = [], allowedPorts = [80, 443], lookupImpl = dns.lookup } = {}) {
+ const exempt = new Set(allowPrivateHosts.map(h => h.toLowerCase().replace(/^\[|\]$/g, '')));
+
+ /** Synchronous URL checks: scheme, credentials, port, IP literal. Throws GuardError. */
+ function checkUrl(input) {
+ let url;
+ try { url = new URL(input); } catch { throw new GuardError('bad_url', 'not a URL'); }
+ if (url.protocol !== 'http:' && url.protocol !== 'https:') throw new GuardError('bad_url', `scheme ${url.protocol} is not fetched`);
+ if (url.username || url.password) throw new GuardError('bad_url', 'credentials in URLs are not allowed');
+ const host = url.hostname.toLowerCase().replace(/^\[|\]$/g, '');
+ const port = url.port ? Number(url.port) : (url.protocol === 'https:' ? 443 : 80);
+ if (!exempt.has(host) && !allowedPorts.includes(port)) throw new GuardError('port_refused', `port ${port} is not allowed`);
+ if (net.isIP(host) && !exempt.has(host) && !isPublicAddress(host)) throw new GuardError('address_refused', `${host} is not a public address`);
+ return url;
+ }
+
+ /** dns.lookup replacement for http(s).request: refuses non-public answers at connect time. */
+ function lookup(hostname, options, callback) {
+ if (typeof options === 'function') { callback = options; options = {}; }
+ const opts = typeof options === 'number' ? { family: options } : { ...(options || {}) };
+ const host = String(hostname).toLowerCase();
+ lookupImpl(hostname, { ...opts, all: true }, (err, addresses) => {
+ if (err) return callback(err);
+ const list = Array.isArray(addresses) ? addresses : [{ address: addresses, family: net.isIP(addresses) }];
+ if (!list.length) return callback(Object.assign(new Error(`no address for ${hostname}`), { code: 'ENOTFOUND' }));
+ if (!exempt.has(host)) {
+ const bad = list.find(a => !isPublicAddress(a.address));
+ if (bad) {
+ const e = new GuardError('address_refused', `${hostname} resolves to a non-public address`);
+ e.errno = 'EADDRREFUSED';
+ return callback(e);
+ }
+ }
+ if (opts.all) return callback(null, list);
+ return callback(null, list[0].address, list[0].family);
+ });
+ }
+
+ return { checkUrl, lookup, isPublicAddress };
+}
+
+module.exports = { createGuard, isPublicAddress, GuardError, embeddedV4 };
new file mode 100644
new file mode 100644
@@ -0,0 +1,240 @@
@@ -0,0 +1,240 @@
+'use strict';
+/**
+ * The source registry: validation, CRUD, and the health/staleness view.
+ *
+ * A source records where it is fetched from and on what terms: endpoints, the auth mode with the
+ * NAME of the environment variable holding the credential (never its value; names must start with
+ * SOURCES_CRED_ so a registry entry can never point at this service's own secrets), robots and
+ * terms notes, the rate limit, whether it is enabled, review/sensitivity flags and the default
+ * indexability downstream products start from. A source cannot be enabled without a terms note:
+ * an adapter existing is not permission to ingest (roadmap anti-goal 26).
+ */
+const { checkApiMapping } = require('./adapters');
+
+const TYPES = ['rss', 'atom', 'sitemap', 'jsonld', 'api', 'manual'];
+const CATEGORIES = ['news', 'blog', 'reviews', 'deals', 'coupons', 'trade'];
+const SENSITIVITY = ['none', 'financial', 'health', 'political', 'legal', 'adult'];
+const AUTH_MODES = ['none', 'header', 'bearer', 'query'];
+const KEY_RE = /^[a-z0-9][a-z0-9-]{1,63}$/;
+const ENV_RE = /^SOURCES_CRED_[A-Z0-9_]{2,60}$/;
+const HEADER_RE = /^[A-Za-z][A-Za-z0-9-]{0,63}$/;
+const FORBIDDEN_HEADERS = new Set(['host', 'cookie', 'connection', 'content-length', 'transfer-encoding', 'user-agent', 'accept-encoding', 'if-none-match', 'if-modified-since']);
+
+class RegistryError extends Error {
+ constructor(detail, code = 'sources.bad_source') { super(detail); this.code = code; }
+}
+
+function httpUrl(v, what) {
+ let u;
+ try { u = new URL(String(v)); } catch { throw new RegistryError(`${what} is not a URL`); }
+ if (u.protocol !== 'http:' && u.protocol !== 'https:') throw new RegistryError(`${what} must be http(s)`);
+ if (u.username || u.password) throw new RegistryError(`${what} must not carry credentials`);
+ if (String(v).length > 2048) throw new RegistryError(`${what} is too long`);
+ return u.toString();
+}
+
+const text = (v, max, what) => {
+ if (v == null || v === '') return null;
+ if (typeof v !== 'string' || v.length > max) throw new RegistryError(`${what} must be a string of at most ${max} characters`);
+ return v.trim() || null;
+};
+const intIn = (v, min, max, d, what) => {
+ if (v === undefined || v === null) return d;
+ if (!Number.isInteger(v) || v < min || v > max) throw new RegistryError(`${what} must be an integer between ${min} and ${max}`);
+ return v;
+};
+const bool = (v, d, what) => {
+ if (v === undefined || v === null) return d;
+ if (typeof v !== 'boolean') throw new RegistryError(`${what} must be true or false`);
+ return v;
+};
+
+/** Validate a full source definition → normalized record (no runtime state). */
+function validateSource(input, { maxItemsCap = 500 } = {}) {
+ if (!input || typeof input !== 'object' || Array.isArray(input)) throw new RegistryError('body must be a source object');
+ const known = new Set(['key', 'name', 'type', 'category', 'homepage_url', 'endpoints', 'auth', 'robots_note', 'terms_note', 'license_note',
+ 'min_interval_ms', 'poll_interval_sec', 'stale_after_sec', 'max_items', 'enabled', 'review_required', 'sensitivity',
+ 'default_indexability', 'search_visibility']);
+ for (const k of Object.keys(input)) if (!known.has(k)) throw new RegistryError(`unknown field ${k}`);
+ if (!KEY_RE.test(String(input.key || ''))) throw new RegistryError('key must match ^[a-z0-9][a-z0-9-]{1,63}$');
+ const name = text(input.name, 200, 'name');
+ if (!name) throw new RegistryError('name is required');
+ if (!TYPES.includes(input.type)) throw new RegistryError(`type must be one of ${TYPES.join('|')}`);
+ if (!CATEGORIES.includes(input.category)) throw new RegistryError(`category must be one of ${CATEGORIES.join('|')}`);
+
+ const endpoints = input.endpoints == null ? [] : input.endpoints;
+ if (!Array.isArray(endpoints) || endpoints.length > 20) throw new RegistryError('endpoints must be an array of at most 20');
+ if (input.type === 'manual' && endpoints.length) throw new RegistryError('a manual source has no endpoints');
+ if (input.type !== 'manual' && !endpoints.length) throw new RegistryError('at least one endpoint is required');
+ const eps = endpoints.map((ep, i) => {
+ const e = typeof ep === 'string' ? { url: ep } : ep;
+ if (!e || typeof e !== 'object') throw new RegistryError(`endpoints[${i}] must be a URL or an object`);
+ const out = { url: httpUrl(e.url, `endpoints[${i}].url`) };
+ if (input.type === 'api') {
+ const err = checkApiMapping(e);
+ if (err) throw new RegistryError(`endpoints[${i}]: ${err}`);
+ Object.assign(out, { format: e.format || 'json', items_path: e.items_path || '', item_kind: e.item_kind || 'record', fields: e.fields || {}, extra: e.extra || {} });
+ } else {
+ for (const k of Object.keys(e)) if (k !== 'url') throw new RegistryError(`endpoints[${i}].${k} is only for api sources`);
+ }
+ return out;
+ });
+ if (new Set(eps.map(e => e.url)).size !== eps.length) throw new RegistryError('endpoints must be distinct');
+
+ const a = input.auth == null ? { mode: 'none' } : input.auth;
+ if (typeof a !== 'object' || !AUTH_MODES.includes(a.mode)) throw new RegistryError(`auth.mode must be one of ${AUTH_MODES.join('|')}`);
+ const auth = { mode: a.mode };
+ if (a.mode !== 'none') {
+ if (input.type === 'manual') throw new RegistryError('a manual source has no credential');
+ if (!ENV_RE.test(String(a.env || ''))) throw new RegistryError('auth.env must be an environment variable NAME matching ^SOURCES_CRED_[A-Z0-9_]+$ (never a value)');
+ auth.env = a.env;
+ if (a.mode === 'header') {
+ if (!HEADER_RE.test(String(a.header || '')) || FORBIDDEN_HEADERS.has(String(a.header).toLowerCase())) throw new RegistryError('auth.header must be a plain header name');
+ auth.header = a.header;
+ }
+ if (a.mode === 'query') {
+ if (!/^[A-Za-z0-9_.-]{1,64}$/.test(String(a.param || ''))) throw new RegistryError('auth.param must be a query parameter name');
+ auth.param = a.param;
+ }
+ }
+ for (const k of Object.keys(a)) if (!['mode', 'env', 'header', 'param'].includes(k)) throw new RegistryError(`unknown auth field ${k}`);
+
+ const pollIntervalSec = intIn(input.poll_interval_sec, 1, 7 * 86400, 3600, 'poll_interval_sec');
+ const record = {
+ key: input.key,
+ name,
+ type: input.type,
+ category: input.category,
+ homepage_url: input.homepage_url ? httpUrl(input.homepage_url, 'homepage_url') : null,
+ endpoints: eps,
+ auth,
+ robots_note: text(input.robots_note, 2000, 'robots_note'),
+ terms_note: text(input.terms_note, 2000, 'terms_note'),
+ license_note: text(input.license_note, 2000, 'license_note'),
+ min_interval_ms: intIn(input.min_interval_ms, 0, 86400000, 60000, 'min_interval_ms'),
+ poll_interval_sec: pollIntervalSec,
+ stale_after_sec: intIn(input.stale_after_sec, 1, 90 * 86400, pollIntervalSec * 3, 'stale_after_sec'),
+ max_items: intIn(input.max_items, 1, maxItemsCap, Math.min(200, maxItemsCap), 'max_items'),
+ enabled: bool(input.enabled, false, 'enabled'),
+ review_required: bool(input.review_required, true, 'review_required'),
+ sensitivity: input.sensitivity == null ? 'none' : input.sensitivity,
+ default_indexability: input.default_indexability == null ? 'noindex' : input.default_indexability,
+ search_visibility: input.search_visibility == null ? null : input.search_visibility,
+ };
+ if (!SENSITIVITY.includes(record.sensitivity)) throw new RegistryError(`sensitivity must be one of ${SENSITIVITY.join('|')}`);
+ if (!['index', 'noindex'].includes(record.default_indexability)) throw new RegistryError('default_indexability must be index or noindex');
+ if (record.sensitivity !== 'none' && record.default_indexability === 'index' && record.review_required === false) {
+ throw new RegistryError('a sensitive source needs review before anything from it can be indexable');
+ }
+ if (![null, 'members'].includes(record.search_visibility)) throw new RegistryError('search_visibility must be null or "members" (staff-only search of raw items)');
+ if (record.enabled && !record.terms_note) throw new RegistryError('a source cannot be enabled without a terms_note (access and terms must be verified first)', 'sources.terms_required');
+ if (record.enabled && record.type !== 'manual' && !record.robots_note) throw new RegistryError('a fetched source cannot be enabled without a robots_note', 'sources.terms_required');
+ return record;
+}
+
+function createRegistry({ db, now = () => Date.now(), maxItemsCap = 500 }) {
+ const st = {
+ get: db.prepare('SELECT * FROM sources WHERE key = ?'),
+ all: db.prepare('SELECT * FROM sources ORDER BY key'),
+ insert: db.prepare(`INSERT INTO sources (key, name, type, category, homepage_url, endpoints, auth, robots_note, terms_note, license_note,
+ min_interval_ms, poll_interval_sec, stale_after_sec, max_items, enabled, review_required, sensitivity, default_indexability,
+ search_visibility, created_at, updated_at, updated_by, next_due_at)
+ VALUES (@key, @name, @type, @category, @homepage_url, @endpoints, @auth, @robots_note, @terms_note, @license_note,
+ @min_interval_ms, @poll_interval_sec, @stale_after_sec, @max_items, @enabled, @review_required, @sensitivity, @default_indexability,
+ @search_visibility, @created_at, @updated_at, @updated_by, 0)`),
+ update: db.prepare(`UPDATE sources SET name = @name, type = @type, category = @category, homepage_url = @homepage_url, endpoints = @endpoints,
+ auth = @auth, robots_note = @robots_note, terms_note = @terms_note, license_note = @license_note, min_interval_ms = @min_interval_ms,
+ poll_interval_sec = @poll_interval_sec, stale_after_sec = @stale_after_sec, max_items = @max_items, enabled = @enabled,
+ review_required = @review_required, sensitivity = @sensitivity, default_indexability = @default_indexability,
+ search_visibility = @search_visibility, updated_at = @updated_at, updated_by = @updated_by WHERE key = @key`),
+ del: db.prepare('DELETE FROM sources WHERE key = ?'),
+ itemCount: db.prepare('SELECT COUNT(*) AS n FROM items WHERE source_key = ?'),
+ };
+
+ function toRow(rec, by) {
+ return {
+ ...rec,
+ endpoints: JSON.stringify(rec.endpoints),
+ auth: JSON.stringify(rec.auth),
+ enabled: rec.enabled ? 1 : 0,
+ review_required: rec.review_required ? 1 : 0,
+ created_at: now(),
+ updated_at: now(),
+ updated_by: by || null,
+ };
+ }
+
+ function fromRow(row) {
+ if (!row) return null;
+ return {
+ key: row.key, name: row.name, type: row.type, category: row.category, homepage_url: row.homepage_url,
+ endpoints: JSON.parse(row.endpoints), auth: JSON.parse(row.auth),
+ robots_note: row.robots_note, terms_note: row.terms_note, license_note: row.license_note,
+ min_interval_ms: row.min_interval_ms, poll_interval_sec: row.poll_interval_sec, stale_after_sec: row.stale_after_sec,
+ max_items: row.max_items, enabled: Boolean(row.enabled), review_required: Boolean(row.review_required),
+ sensitivity: row.sensitivity, default_indexability: row.default_indexability, search_visibility: row.search_visibility,
+ };
+ }
+
+ /** Health and staleness, computed from the runtime columns. */
+ function health(row) {
+ const t = now();
+ const neverSucceeded = row.last_success_at == null;
+ const stale = row.type === 'manual' ? false : neverSucceeded || (t - row.last_success_at) > row.stale_after_sec * 1000;
+ let status;
+ if (!row.enabled) status = 'disabled';
+ else if (row.type === 'manual') status = 'manual';
+ else if (row.last_run_at == null) status = 'never_fetched';
+ else if (row.consecutive_failures > 0) status = 'failing';
+ else if (stale) status = 'stale';
+ else status = 'healthy';
+ const iso = (v) => (v == null ? null : new Date(v).toISOString());
+ return {
+ status,
+ stale,
+ last_success_at: iso(row.last_success_at),
+ last_run_at: iso(row.last_run_at),
+ last_state: row.last_state,
+ consecutive_failures: row.consecutive_failures,
+ next_due_at: row.enabled && row.type !== 'manual' ? iso(Math.max(row.next_due_at, row.not_before)) : null,
+ stale_after_sec: row.stale_after_sec,
+ };
+ }
+
+ function view(row) {
+ return { ...fromRow(row), health: health(row), created_at: new Date(row.created_at).toISOString(), updated_at: new Date(row.updated_at).toISOString() };
+ }
+
+ function create(input, by) {
+ const rec = validateSource(input, { maxItemsCap });
+ if (st.get.get(rec.key)) throw new RegistryError(`source ${rec.key} exists`, 'sources.exists');
+ st.insert.run(toRow(rec, by));
+ return st.get.get(rec.key);
+ }
+
+ function patch(key, changes, by) {
+ const row = st.get.get(key);
+ if (!row) return null;
+ if (changes && changes.key !== undefined && changes.key !== key) throw new RegistryError('key cannot change');
+ if (changes && changes.type !== undefined && changes.type !== row.type && st.itemCount.get(key).n) {
+ throw new RegistryError('type cannot change once the source has items (register a new source)');
+ }
+ const merged = { ...fromRow(row), ...(changes || {}), key };
+ const rec = validateSource(merged, { maxItemsCap });
+ st.update.run({ ...toRow(rec, by) });
+ return st.get.get(key);
+ }
+
+ /** Only a source that never produced an item can be deleted; otherwise disable it. */
+ function remove(key) {
+ const row = st.get.get(key);
+ if (!row) return null;
+ if (st.itemCount.get(key).n) throw new RegistryError('the source has items; disable it instead (provenance must stay resolvable)', 'sources.has_items');
+ st.del.run(key);
+ return true;
+ }
+
+ return { get: (k) => st.get.get(k), all: () => st.all.all(), create, patch, remove, view, fromRow, health };
+}
+
+module.exports = { createRegistry, validateSource, RegistryError, TYPES, CATEGORIES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,147 @@
@@ -0,0 +1,147 @@
+'use strict';
+/**
+ * robots.txt (RFC 9309), checked before every request an ingestion run makes, including each
+ * redirect hop to a new origin.
+ *
+ * - the group whose User-agent names our product token (case-insensitive) applies, else `*`;
+ * several matching groups are merged
+ * - Allow/Disallow with `*` and `$`; the longest matching rule wins, Allow wins a tie
+ * - 2xx: parsed; 4xx (e.g. 404) except 429: no restrictions; 429, 5xx, network error or timeout:
+ * complete disallow ("unreachable") — never guessed as allowed
+ * - cached per origin for SOURCES_ROBOTS_TTL_MS (24 h); Crawl-delay is honoured as a minimum
+ * gap between requests (non-standard, but a clear wish of the site)
+ * A source-adapter name is never permission to fetch (roadmap anti-goal 26): if robots says no,
+ * the fetch run is `robots_denied` and nothing is requested.
+ */
+
+function parseRobots(text, agent) {
+ const token = String(agent).toLowerCase();
+ const groups = [];
+ let current = null;
+ let lastWasAgent = false;
+ for (const rawLine of String(text).split(/\r\n|\r|\n/)) {
+ const line = rawLine.replace(/#.*$/, '').trim();
+ if (!line) continue;
+ const i = line.indexOf(':');
+ if (i < 0) continue;
+ const key = line.slice(0, i).trim().toLowerCase();
+ const value = line.slice(i + 1).trim();
+ if (key === 'user-agent') {
+ if (!lastWasAgent || !current) {
+ current = { agents: [], rules: [], crawlDelay: null };
+ groups.push(current);
+ }
+ current.agents.push(value.toLowerCase());
+ lastWasAgent = true;
+ continue;
+ }
+ lastWasAgent = false;
+ if (!current) continue;
+ // Bounded: a hostile robots.txt cannot make matching expensive.
+ if ((key === 'allow' || key === 'disallow') && value.length <= 512 && current.rules.length < 1000) {
+ current.rules.push({ allow: key === 'allow', path: value.replace(/\*+/g, '*') });
+ }
+ else if (key === 'crawl-delay') {
+ const n = Number(value);
+ if (Number.isFinite(n) && n >= 0) current.crawlDelay = Math.min(n, 3600);
+ }
+ }
+ // RFC 9309: the product token is matched case-insensitively against the User-agent value.
+ let chosen = groups.filter(g => g.agents.some(a => a.split('/')[0].trim() === token));
+ if (!chosen.length) chosen = groups.filter(g => g.agents.includes('*'));
+ const delays = chosen.map(g => g.crawlDelay).filter(d => d != null);
+ return {
+ // An empty Disallow (or Allow) path matches nothing: "allow everything".
+ rules: chosen.flatMap(g => g.rules).filter(r => r.path !== ''),
+ crawlDelaySec: delays.length ? Math.max(...delays) : null,
+ };
+}
+
+function ruleRegex(path) {
+ let p = path;
+ const anchored = p.endsWith('$');
+ if (anchored) p = p.slice(0, -1);
+ const body = p.split('*').map(s => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');
+ return new RegExp('^' + body + (anchored ? '$' : ''));
+}
+
+function normalizePath(p) {
+ // Compare percent-encoding consistently: decode what is safe to decode, re-encode the rest.
+ try { return encodeURI(decodeURI(p)); } catch { return p; }
+}
+
+/** Is `pathWithQuery` allowed by the parsed rules? */
+function isAllowed(parsed, pathWithQuery) {
+ const target = normalizePath(pathWithQuery || '/');
+ let best = null;
+ for (const r of parsed.rules) {
+ const re = ruleRegex(normalizePath(r.path));
+ if (!re.test(target)) continue;
+ const len = r.path.length;
+ if (!best || len > best.len || (len === best.len && r.allow && !best.allow)) best = { len, allow: r.allow };
+ }
+ return best ? best.allow : true;
+}
+
+/**
+ * createRobots({ db, fetcher, agent, ttlMs, maxBytes, now, beforeRequest(host) })
+ * check(url) → { allowed, reason, crawlDelaySec, source: 'cache'|'fetched' }
+ */
+function createRobots({ db, fetcher, agent, ttlMs, maxBytes, now = () => Date.now(), beforeRequest = async () => {} }) {
+ const get = db.prepare('SELECT * FROM robots_cache WHERE origin = ?');
+ const put = db.prepare(`INSERT INTO robots_cache (origin, fetched_at, expires_at, status, outcome, rules, crawl_delay_sec, detail)
+ VALUES (@origin, @fetched_at, @expires_at, @status, @outcome, @rules, @crawl_delay_sec, @detail)
+ ON CONFLICT(origin) DO UPDATE SET fetched_at = excluded.fetched_at, expires_at = excluded.expires_at, status = excluded.status,
+ outcome = excluded.outcome, rules = excluded.rules, crawl_delay_sec = excluded.crawl_delay_sec, detail = excluded.detail`);
+ const inflight = new Map();
+
+ async function load(origin) {
+ await beforeRequest(new URL(origin).host);
+ const t = now();
+ let row;
+ try {
+ const r = await fetcher.fetchUrl(`${origin}/robots.txt`, { headers: { Accept: 'text/plain' }, capBytes: maxBytes });
+ if (r.status >= 200 && r.status < 300) {
+ const parsed = parseRobots(r.body.toString('utf8'), agent);
+ row = { status: r.status, outcome: 'parsed', rules: JSON.stringify(parsed.rules), crawl_delay_sec: parsed.crawlDelaySec, detail: null };
+ } else if (r.status >= 400 && r.status < 500 && r.status !== 429) {
+ row = { status: r.status, outcome: 'unavailable', rules: '[]', crawl_delay_sec: null, detail: `HTTP ${r.status}: no restrictions` };
+ } else {
+ row = { status: r.status, outcome: 'unreachable', rules: '[]', crawl_delay_sec: null, detail: `HTTP ${r.status}: complete disallow` };
+ }
+ } catch (err) {
+ // Network error, timeout, refused address or an oversized file: refuse, never guess.
+ row = { status: null, outcome: 'unreachable', rules: '[]', crawl_delay_sec: null, detail: `${err.code || 'error'}: ${err.message}`.slice(0, 300) };
+ }
+ // An unreachable robots.txt is retried sooner than a parsed one.
+ const ttl = row.outcome === 'unreachable' ? Math.min(ttlMs, 15 * 60 * 1000) : ttlMs;
+ const full = { origin, fetched_at: t, expires_at: t + ttl, ...row };
+ put.run(full);
+ return full;
+ }
+
+ async function entry(origin) {
+ const cached = get.get(origin);
+ if (cached && cached.expires_at > now()) return { row: cached, source: 'cache' };
+ if (!inflight.has(origin)) inflight.set(origin, load(origin).finally(() => inflight.delete(origin)));
+ return { row: await inflight.get(origin), source: 'fetched' };
+ }
+
+ async function check(input) {
+ const url = new URL(input);
+ const { row, source } = await entry(url.origin);
+ if (row.outcome === 'unreachable') {
+ // A refused address/port is a policy refusal of the host itself, not a robots answer.
+ const code = row.status == null ? String(row.detail || '').split(':')[0] : null;
+ return { allowed: false, reason: `robots.txt unreachable (${row.detail})`, crawlDelaySec: null, source, refusedCode: ['address_refused', 'port_refused', 'bad_url'].includes(code) ? code : null };
+ }
+ if (row.outcome === 'unavailable') return { allowed: true, reason: null, crawlDelaySec: null, source };
+ const parsed = { rules: JSON.parse(row.rules) };
+ const allowed = isAllowed(parsed, url.pathname + url.search);
+ return { allowed, reason: allowed ? null : `disallowed by ${url.origin}/robots.txt`, crawlDelaySec: row.crawl_delay_sec, source };
+ }
+
+ return { check, forget: (origin) => db.prepare('DELETE FROM robots_cache WHERE origin = ?').run(origin) };
+}
+
+module.exports = { parseRobots, isAllowed, createRobots };
new file mode 100644
new file mode 100644
@@ -0,0 +1,43 @@
@@ -0,0 +1,43 @@
+'use strict';
+/**
+ * In-process scheduler: every SOURCES_TICK_MS, start runs for enabled, fetched sources whose
+ * next_due_at and not_before have passed, oldest first, up to SOURCES_MAX_CONCURRENT runs at a
+ * time. ingest.run() itself refuses a second run of a source already in flight.
+ */
+function createScheduler({ db, ingest, config, now = () => Date.now(), log = console }) {
+ const due = db.prepare(`SELECT key FROM sources WHERE enabled = 1 AND type != 'manual'
+ AND next_due_at <= @now AND not_before <= @now ORDER BY next_due_at, key LIMIT @limit`);
+ let timer = null;
+ const running = new Set();
+
+ function tick() {
+ const free = config.worker.maxConcurrent - running.size;
+ if (free <= 0) return [];
+ const keys = due.all({ now: now(), limit: free + running.size }).map(r => r.key).filter(k => !running.has(k)).slice(0, free);
+ for (const key of keys) {
+ running.add(key);
+ ingest.run(key, { trigger: 'schedule' })
+ .catch(err => log.error(`[scheduler] ${key}: ${err.stack || err}`))
+ .finally(() => running.delete(key));
+ }
+ return keys;
+ }
+
+ function start() {
+ if (timer) return;
+ timer = setInterval(() => { try { tick(); } catch (err) { log.error(`[scheduler] tick: ${err.message}`); } }, config.worker.tickMs);
+ timer.unref?.();
+ }
+
+ async function stop() {
+ if (timer) clearInterval(timer);
+ timer = null;
+ // let runs in flight finish (each is bounded by the fetch timeout)
+ const deadline = Date.now() + 15000;
+ while (running.size && Date.now() < deadline) await new Promise(r => setTimeout(r, 50));
+ }
+
+ return { start, stop, tick, running: () => Boolean(timer), active: () => [...running] };
+}
+
+module.exports = { createScheduler };
new file mode 100644
new file mode 100644
@@ -0,0 +1,38 @@
@@ -0,0 +1,38 @@
+'use strict';
+/**
+ * Per-host request spacing shared by robots.txt fetches and ingestion: before any request to a
+ * host, wait until max(SOURCES_HOST_MIN_INTERVAL_MS, the source's min interval, the host's robots
+ * Crawl-delay) has passed since the previous request to it. Requests to one host are serialized.
+ */
+const sleep = (ms) => new Promise(r => setTimeout(r, ms));
+
+function createSpacer({ hostMinIntervalMs = 1000, now = () => Date.now() } = {}) {
+ const hosts = new Map(); // host → { last, crawlDelayMs, chain }
+
+ function state(host) {
+ let h = hosts.get(host);
+ if (!h) { h = { last: 0, crawlDelayMs: 0, chain: Promise.resolve() }; hosts.set(host, h); }
+ return h;
+ }
+
+ function space(host, minMs = 0) {
+ const h = state(host);
+ const turn = h.chain.then(async () => {
+ const gap = Math.max(hostMinIntervalMs, minMs || 0, h.crawlDelayMs);
+ const wait = h.last + gap - now();
+ if (wait > 0) await sleep(wait);
+ h.last = now();
+ });
+ h.chain = turn.catch(() => {});
+ return turn;
+ }
+
+ function setCrawlDelay(host, sec) {
+ if (sec == null) return;
+ state(host).crawlDelayMs = Math.min(sec * 1000, 3600 * 1000);
+ }
+
+ return { space, setCrawlDelay };
+}
+
+module.exports = { createSpacer };
new file mode 100644
new file mode 100644
@@ -0,0 +1,166 @@
@@ -0,0 +1,166 @@
+'use strict';
+/** Parsers against fixtures: what the source states is kept, what it does not state stays null. */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const feed = require('../server/adapters/feed');
+const sitemap = require('../server/adapters/sitemap');
+const jsonld = require('../server/adapters/jsonld');
+const api = require('../server/adapters/api');
+const { decodeBody, toIsoDate, canonicalUrl } = require('../server/adapters/util');
+const { suite } = require('./helpers');
+
+const t = suite('adapters');
+const fx = (name) => fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');
+
+t('RSS 2.0: guid identity, cleaned link, text-only summary, dates, skips', () => {
+ const r = feed.parse(fx('rss2.xml'), { url: 'https://news.example.org/feed.xml' });
+ assert.strictEqual(r.meta.format, 'rss2');
+ assert.strictEqual(r.items.length, 2);
+ assert.strictEqual(r.skipped, 2, 'the identity-less item and the duplicate guid');
+ const [a, b] = r.items;
+ assert.strictEqual(a.identity, 'news-0001');
+ assert.strictEqual(a.canonical_url, 'https://news.example.org/2026/09/first-light?id=7', 'tracking params and fragment removed');
+ assert.strictEqual(a.title, 'Telescope sees first light');
+ assert.strictEqual(a.summary, 'The new telescope & its camera saw first light .');
+ assert.ok(!a.summary.includes('alert'), 'scripts are dropped');
+ assert.strictEqual(a.published_at, '2026-09-21T14:30:00.000Z');
+ assert.deepStrictEqual(a.authors, ['Ada Example']);
+ assert.deepStrictEqual(a.fields.categories, ['Astronomy', 'Instruments']);
+ assert.deepStrictEqual(a.fields.enclosure, { url: 'https://news.example.org/img/first-light.jpg', type: 'image/jpeg', length: 12345 });
+ assert.strictEqual(b.identity, 'https://news.example.org/2026/09/undated');
+ assert.strictEqual(b.published_at, null, 'no date in the feed = no date, never "now"');
+ assert.deepStrictEqual(b.authors, []);
+});
+
+t('Atom: id identity, relative alternate link, html title, bad dates are null', () => {
+ const r = feed.parse(fx('atom.xml'), { url: 'https://blog.example.com/feed.atom' });
+ assert.strictEqual(r.meta.format, 'atom');
+ assert.strictEqual(r.items.length, 2);
+ const [a, b] = r.items;
+ assert.strictEqual(a.identity, 'tag:blog.example.com,2026:posts/release-2');
+ assert.strictEqual(a.canonical_url, 'https://blog.example.com/posts/release-2');
+ assert.strictEqual(a.title, 'Release 2.0 notes');
+ assert.strictEqual(a.published_at, '2026-09-19T10:00:00.000Z');
+ assert.strictEqual(a.updated_at, '2026-09-20T18:30:02.000Z');
+ assert.deepStrictEqual(a.authors, ['Grace Example']);
+ assert.deepStrictEqual(a.fields.categories, ['releases']);
+ assert.strictEqual(b.updated_at, null);
+ assert.strictEqual(b.summary, 'Body');
+});
+
+t('a large real-world-sized feed with thousands of entities parses (no expansion limit tripped)', () => {
+ const items = Array.from({ length: 1500 }, (_, i) => `<item><title>Deal ${i} & more — $${i}</title><link>https://deals.example.com/d/${i}?a=1&b=2</link><guid>d-${i}</guid></item>`).join('');
+ const r = feed.parse(`<?xml version="1.0"?><rss version="2.0"><channel><title>Big & busy</title>${items}</channel></rss>`, { url: 'https://deals.example.com/rss', maxItems: 2000 });
+ assert.strictEqual(r.items.length, 1500);
+ assert.strictEqual(r.items[7].title, 'Deal 7 & more — $7');
+ assert.strictEqual(r.items[7].canonical_url, 'https://deals.example.com/d/7?a=1&b=2');
+});
+
+t('RSS 1.0 (RDF)', () => {
+ const r = feed.parse(fx('rss1.rdf'), { url: 'https://rdf.example.net/index.rdf' });
+ assert.strictEqual(r.meta.format, 'rss1');
+ assert.strictEqual(r.items[0].identity, 'https://rdf.example.net/a');
+ assert.strictEqual(r.items[0].published_at, '2026-09-18T08:00:00.000Z');
+});
+
+t('XML with entity declarations is refused; garbage is a parse error', () => {
+ assert.throws(() => feed.parse(fx('xxe.xml'), { url: 'https://x.example/' }), (e) => e.code === 'parse_error' && /entity/.test(e.message));
+ assert.throws(() => feed.parse('<html><body>not a feed</body></html>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');
+ assert.throws(() => feed.parse('<rss><channel><item>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');
+});
+
+t('sitemap urlset with the news extension; index lists children without following them', () => {
+ const r = sitemap.parse(fx('sitemap.xml'), { url: 'https://deals.example.com/sitemap.xml' });
+ assert.strictEqual(r.items.length, 2);
+ assert.strictEqual(r.skipped, 1, 'a non-http loc');
+ assert.strictEqual(r.items[0].canonical_url, 'https://deals.example.com/offers/1');
+ assert.strictEqual(r.items[0].updated_at, '2026-09-21T00:00:00.000Z');
+ assert.strictEqual(r.items[0].fields.priority, 0.8);
+ assert.strictEqual(r.items[0].title, null);
+ assert.strictEqual(r.items[1].title, 'Launch week');
+ assert.strictEqual(r.items[1].published_at, '2026-09-20T09:00:00.000Z');
+ assert.strictEqual(r.items[1].fields.publication, 'Deals Example');
+ const idx = sitemap.parse(fx('sitemap-index.xml'), { url: 'https://deals.example.com/sitemap-index.xml' });
+ assert.deepStrictEqual(idx.items.map(i => [i.kind, i.canonical_url]), [
+ ['sitemap', 'https://deals.example.com/sitemap-1.xml'], ['sitemap', 'https://deals.example.com/sitemap-2.xml'],
+ ]);
+});
+
+t('JSON-LD Product: offers as stated, missing price stays null, rating copied not computed, review text not kept', () => {
+ const r = jsonld.parse(fx('product.html'), { url: 'https://shop.example.com/p/widget-pro' });
+ assert.strictEqual(r.items.length, 1, 'BreadcrumbList and Organization are not items');
+ const p = r.items[0];
+ assert.strictEqual(p.kind, 'product');
+ assert.strictEqual(p.identity, 'https://shop.example.com/p/widget-pro');
+ assert.strictEqual(p.title, 'Widget Pro');
+ assert.strictEqual(p.summary, 'A very good widget.');
+ assert.strictEqual(p.fields.sku, 'WP-100');
+ assert.strictEqual(p.fields.gtin, '0012345678905');
+ assert.strictEqual(p.fields.brand, 'Example Co');
+ assert.deepStrictEqual(p.fields.offers[0], {
+ type: 'Offer', price: '19.99', low_price: null, high_price: null, offer_count: null, currency: 'USD', availability: 'InStock',
+ condition: null, valid_until: '2026-12-31T00:00:00.000Z', url: 'https://shop.example.com/p/widget-pro', seller: 'Example Shop',
+ });
+ assert.strictEqual(p.fields.offers[1].price, null, 'no price on the page = null, never 0 or a guess');
+ assert.strictEqual(p.fields.offers[1].availability, 'OutOfStock');
+ assert.deepStrictEqual(p.fields.aggregate_rating, { value: 4.4, best: null, worst: null, count: null, review_count: 89 });
+ assert.deepStrictEqual(p.fields.reviews, [{ author: 'Sam', rating: { value: 5, best: 5, worst: null, count: null, review_count: null }, date: '2026-08-01T00:00:00.000Z' }]);
+ assert.ok(!JSON.stringify(p).includes('Long text'), 'embedded review bodies are not stored');
+});
+
+t('JSON-LD NewsArticle, Review and BlogPosting; an invalid block is skipped, not fatal', () => {
+ const r = jsonld.parse(fx('article.html'), { url: 'https://times.example.org/2026/09/budget' });
+ assert.deepStrictEqual(r.items.map(i => i.kind), ['article', 'review', 'article']);
+ const [news, review, blog] = r.items;
+ assert.strictEqual(news.title, 'Council approves budget');
+ assert.strictEqual(news.canonical_url, 'https://times.example.org/2026/09/budget');
+ assert.strictEqual(news.published_at, '2026-09-21T05:00:00.000Z');
+ assert.deepStrictEqual(news.authors, ['Lee Reporter']);
+ assert.strictEqual(news.fields.publisher, 'Example Times');
+ assert.strictEqual(review.fields.item_reviewed, 'Widget Pro');
+ assert.deepStrictEqual(review.fields.rating, { value: 3, best: 5, worst: null, count: null, review_count: null });
+ assert.strictEqual(review.summary, 'Solid, not spectacular.');
+ assert.strictEqual(blog.identity, 'https://times.example.org/blog/why-widgets');
+ assert.strictEqual(r.skipped, 1, 'the invalid JSON block');
+ assert.notStrictEqual(news.identity, review.identity);
+});
+
+t('a page without JSON-LD is a parse error, not an empty success', () => {
+ assert.throws(() => jsonld.parse(fx('no-jsonld.html'), { url: 'https://x.example/' }), (e) => e.code === 'parse_error');
+ assert.throws(() => jsonld.parse('<script type="application/ld+json">nope</script>', { url: 'https://x.example/' }), (e) => e.code === 'parse_error');
+});
+
+t('API mapping: fields by path, epoch dates, nulls kept, identity-less records skipped', () => {
+ const endpoint = {
+ format: 'json', items_path: 'deals', item_kind: 'offer',
+ fields: { identity: 'dealID', url: 'link', title: 'title', published_at_epoch: 'releaseDate' },
+ extra: { sale_price: 'salePrice', normal_price: 'normalPrice', store_id: 'storeID' },
+ };
+ assert.strictEqual(api.checkMapping(endpoint), null);
+ const r = api.parse(fx('deals.json'), { url: 'https://api.example.com/deals', endpoint });
+ assert.strictEqual(r.items.length, 2);
+ assert.strictEqual(r.skipped, 1);
+ assert.deepStrictEqual(r.items[0].fields, { sale_price: '4.99', normal_price: '19.99', store_id: '1' });
+ assert.strictEqual(r.items[0].published_at, '2024-09-20T00:00:00.000Z');
+ assert.strictEqual(r.items[1].fields.sale_price, null);
+ assert.strictEqual(r.items[1].canonical_url, null);
+ assert.strictEqual(r.items[1].published_at, null);
+ assert.throws(() => api.parse('{"deals": {}}', { url: 'https://api.example.com/', endpoint }), (e) => e.code === 'parse_error');
+ assert.throws(() => api.parse('not json', { url: 'https://api.example.com/', endpoint }), (e) => e.code === 'parse_error');
+ assert.match(api.checkMapping({ fields: { title: 'a' } }), /identity/);
+ assert.match(api.checkMapping({ fields: { identity: 'a b' } }), /malformed/);
+});
+
+t('helpers: charset decoding, date and URL strictness', () => {
+ const latin = Buffer.from('<?xml version="1.0" encoding="ISO-8859-1"?><x>caf\xe9</x>', 'latin1');
+ assert.ok(decodeBody(latin, 'text/xml').includes('café'));
+ assert.strictEqual(toIsoDate('1726790400'), null, 'bare numbers are ambiguous');
+ assert.strictEqual(toIsoDate('yesterday'), null);
+ assert.strictEqual(toIsoDate('1970-01-01'), null, 'implausible years are refused');
+ assert.strictEqual(canonicalUrl('javascript:alert(1)'), null);
+ assert.strictEqual(canonicalUrl('https://user:[email protected]/'), null);
+ assert.strictEqual(canonicalUrl('HTTPS://Example.COM:443/a?gclid=1&b=2#x'), 'https://example.com/a?b=2');
+});
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,155 @@
@@ -0,0 +1,155 @@
+'use strict';
+/** Read and admin API: capabilities, registry validation, items paging, manual items, removal, health. */
+const assert = require('assert');
+const { boot, request, serviceToken, site, sourceDef, rss, suite } = require('./helpers');
+
+const t = suite('api');
+let svc;
+let web;
+const ADMIN = serviceToken('network', ['sources.source.manage']);
+const READER = serviceToken('news', ['sources.source.read', 'sources.item.read']);
+const ITEMS_ONLY = serviceToken('deals', ['sources.item.read']);
+
+const api = (method, p, opts = {}) => request(svc.base, method, p, opts);
+
+t('boot', async () => {
+ svc = await boot();
+ web = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/feed.xml': () => ({ body: rss([1, 2, 3, 4, 5].map(i => ({ guid: `g${i}`, title: `Item ${i}`, link: `https://example.org/${i}` }))) }),
+ });
+});
+
+t('every route needs a service token with its one capability', async () => {
+ assert.strictEqual((await api('GET', '/api/v1/sources')).status, 401);
+ assert.strictEqual((await api('GET', '/api/v1/sources', { token: ITEMS_ONLY })).status, 403);
+ assert.strictEqual((await api('GET', '/api/v1/items', { token: serviceToken('news', ['sources.source.read']) })).status, 403);
+ assert.strictEqual((await api('POST', '/api/v1/sources', { token: READER, body: sourceDef() })).status, 403);
+ assert.strictEqual((await api('GET', '/api/v1/sources', { token: serviceToken('news', ['sources.source.read'], { aud: 'openvibe.search' }) })).status, 401);
+ assert.strictEqual((await api('GET', '/api/v1/sources', { token: serviceToken('news', ['sources.*']) })).status, 200, 'a family grant covers it');
+});
+
+t('registry validation: terms before enabling, credentials by SOURCES_CRED_ name only, sane types', async () => {
+ const bad = async (def, re) => {
+ const r = await api('POST', '/api/v1/sources', { token: ADMIN, body: def });
+ assert.strictEqual(r.status, 422, JSON.stringify(def));
+ assert.match(r.body.detail, re);
+ };
+ await bad(sourceDef({ terms_note: undefined, endpoints: ['https://example.org/feed'] }), /terms_note/);
+ await bad(sourceDef({ robots_note: undefined, endpoints: ['https://example.org/feed'] }), /robots_note/);
+ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'bearer', env: 'OV_OAUTH_CLIENT_SECRET' } }), /SOURCES_CRED_/);
+ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'bearer', env: 'SOURCES_CRED_XX', value: 'secret' } }), /unknown auth field/);
+ await bad(sourceDef({ endpoints: ['https://example.org/feed'], auth: { mode: 'header', env: 'SOURCES_CRED_XX', header: 'Cookie' } }), /header/);
+ await bad(sourceDef({ endpoints: ['file:///etc/passwd'] }), /http/);
+ await bad(sourceDef({ endpoints: [] }), /endpoint/);
+ await bad(sourceDef({ type: 'manual', endpoints: ['https://example.org/'] }), /manual/);
+ await bad(sourceDef({ category: 'gossip', endpoints: ['https://example.org/'] }), /category/);
+ await bad(sourceDef({ type: 'api', endpoints: [{ url: 'https://api.example.org/', fields: { title: 'x' } }] }), /identity/);
+ await bad(sourceDef({ endpoints: ['https://example.org/'], sensitivity: 'financial', default_indexability: 'index', review_required: false }), /review/);
+ // disabled without notes is fine: a proposal waiting for review
+ const draft = await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'proposal', name: 'Proposal', type: 'rss', category: 'blog', endpoints: ['https://example.org/feed'] } });
+ assert.strictEqual(draft.status, 201);
+ assert.deepStrictEqual([draft.body.source.enabled, draft.body.source.review_required, draft.body.source.default_indexability], [false, true, 'noindex']);
+ assert.strictEqual(draft.body.source.health.status, 'disabled');
+ assert.strictEqual((await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'proposal', name: 'x', type: 'rss', category: 'blog', endpoints: ['https://example.org/f'] } })).status, 409);
+ const enable = await api('PATCH', '/api/v1/sources/proposal', { token: ADMIN, body: { enabled: true } });
+ assert.strictEqual(enable.status, 422);
+ assert.strictEqual(enable.body.code, 'sources.terms_required');
+});
+
+t('create, read, fetch, list runs; views never carry a credential value', async () => {
+ const def = sourceDef({ key: 'feed-a', endpoints: [`${web.origin}/feed.xml`], auth: { mode: 'query', env: 'SOURCES_CRED_FEED_A', param: 'key' } });
+ const c = await api('POST', '/api/v1/sources', { token: ADMIN, body: def });
+ assert.strictEqual(c.status, 201);
+ assert.deepStrictEqual(c.body.source.auth, { mode: 'query', env: 'SOURCES_CRED_FEED_A', param: 'key' });
+ // the variable is not set in this process: the fetch is recorded as disabled
+ let f = await api('POST', '/api/v1/sources/feed-a/fetch', { token: ADMIN });
+ assert.strictEqual(f.body.runs[0].state, 'disabled');
+ await api('PATCH', '/api/v1/sources/feed-a', { token: ADMIN, body: { auth: { mode: 'none' } } });
+ svc.db.prepare('UPDATE sources SET last_request_at = NULL').run();
+ f = await api('POST', '/api/v1/sources/feed-a/fetch', { token: ADMIN });
+ assert.strictEqual(f.status, 200);
+ assert.deepStrictEqual(f.body.runs.map(r => [r.state, r.items.created]), [['ok', 5]]);
+ const runs = await api('GET', '/api/v1/sources/feed-a/runs', { token: READER });
+ assert.deepStrictEqual(runs.body.runs.map(r => r.state), ['ok', 'disabled']);
+ const all = await api('GET', '/api/v1/runs?state=failed', { token: READER });
+ assert.strictEqual(all.status, 200);
+ const one = await api('GET', '/api/v1/sources/feed-a', { token: READER });
+ assert.strictEqual(one.body.source.health.status, 'healthy');
+ assert.strictEqual((await api('GET', '/api/v1/sources/nope', { token: READER })).status, 404);
+});
+
+t('items page in change order and carry provenance and source staleness', async () => {
+ const p1 = await api('GET', '/api/v1/items?source=feed-a&limit=2', { token: ITEMS_ONLY });
+ assert.strictEqual(p1.status, 200);
+ assert.strictEqual(p1.body.items.length, 2);
+ assert.strictEqual(p1.body.more, true);
+ const p2 = await api('GET', `/api/v1/items?source=feed-a&limit=10&after=${p1.body.next_after}`, { token: ITEMS_ONLY });
+ assert.strictEqual(p2.body.items.length, 3);
+ assert.strictEqual(p2.body.more, false);
+ const it = p1.body.items[0];
+ assert.ok(it.provenance.retrieved_at && it.provenance.content_hash && it.provenance.raw_body_hash && it.provenance.parser_version === 'feed@1');
+ assert.strictEqual(it.provenance.terms_note, 'test fixture');
+ assert.deepStrictEqual(Object.keys(p1.body.sources), ['feed-a']);
+ assert.strictEqual(p1.body.sources['feed-a'].stale, false);
+ const byCat = await api('GET', '/api/v1/items?category=news', { token: ITEMS_ONLY });
+ assert.strictEqual(byCat.body.items.length, 5);
+ assert.strictEqual((await api('GET', '/api/v1/items?category=gossip', { token: ITEMS_ONLY })).status, 400);
+ // resume after the last page: nothing new yet
+ const tail = await api('GET', `/api/v1/items?source=feed-a&after=${p2.body.next_after}`, { token: ITEMS_ONLY });
+ assert.deepStrictEqual([tail.body.items.length, tail.body.next_after], [0, p2.body.next_after]);
+});
+
+t('removal needs a reason, is visible with include_removed, and appears in the change feed', async () => {
+ const list = await api('GET', '/api/v1/items?source=feed-a', { token: ITEMS_ONLY });
+ const target = list.body.items[0];
+ const last = list.body.next_after;
+ assert.strictEqual((await api('DELETE', `/api/v1/items/${target.id}`, { token: ADMIN, body: {} })).status, 422);
+ const del = await api('DELETE', `/api/v1/items/${target.id}`, { token: ADMIN, body: { reason: 'licence withdrawn' } });
+ assert.strictEqual(del.status, 200);
+ assert.match(del.body.item.removed.reason, /licence withdrawn \(by svc:network\)/);
+ const feed = await api('GET', `/api/v1/items?source=feed-a&after=${last}&include_removed=1`, { token: ITEMS_ONLY });
+ assert.deepStrictEqual(feed.body.items.map(i => [i.id, Boolean(i.removed)]), [[target.id, true]]);
+ const hidden = await api('GET', `/api/v1/items?source=feed-a&after=${last}`, { token: ITEMS_ONLY });
+ assert.strictEqual(hidden.body.items.length, 0);
+ const one = await api('GET', `/api/v1/items/${target.id}?revisions=1`, { token: ITEMS_ONLY });
+ assert.deepStrictEqual(one.body.item.revisions.map(r => r.revision), [1, 2]);
+ assert.strictEqual((await api('DELETE', '/api/v1/sources/feed-a', { token: ADMIN })).status, 409, 'a source with items cannot be deleted');
+ const gone = await api('DELETE', '/api/v1/sources/proposal', { token: ADMIN });
+ assert.strictEqual(gone.status, 204, gone.text + JSON.stringify(svc.db.prepare('SELECT key FROM sources').all()));
+});
+
+t('manual sources: items need an evidence URL; fetches are refused', async () => {
+ const m = await api('POST', '/api/v1/sources', { token: ADMIN, body: { key: 'staff-coupons', name: 'Staff-entered codes', type: 'manual', category: 'coupons', terms_note: 'codes the merchant published itself', enabled: true } });
+ assert.strictEqual(m.status, 201);
+ assert.strictEqual(m.body.source.health.status, 'manual');
+ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/fetch', { token: ADMIN })).status, 409);
+ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10' } })).status, 422);
+ assert.strictEqual((await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10', url: 'https://merchant.example/promo', published_at: 'soon' } })).status, 422);
+ const add = await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: {
+ title: 'SAVE10', url: 'https://merchant.example/promo?utm_source=x', kind: 'coupon', fields: { code: 'SAVE10', expires: null },
+ } });
+ assert.strictEqual(add.status, 201);
+ assert.strictEqual(add.body.item.canonical_url, 'https://merchant.example/promo');
+ assert.strictEqual(add.body.item.fields.expires, null, 'unknown expiry stays unknown');
+ assert.strictEqual(add.body.item.provenance.entered_by, 'svc:network');
+ assert.strictEqual(add.body.item.provenance.parser_version, 'manual@1');
+ const again = await api('POST', '/api/v1/sources/staff-coupons/items', { token: ADMIN, body: { title: 'SAVE10', url: 'https://merchant.example/promo', kind: 'coupon', fields: { code: 'SAVE10', expires: null } } });
+ assert.strictEqual(again.body.outcome, 'unchanged');
+ assert.strictEqual((await api('POST', '/api/v1/sources/feed-a/items', { token: ADMIN, body: { title: 'x', url: 'https://x.example/' } })).status, 409);
+});
+
+t('health lists what needs attention', async () => {
+ await api('POST', '/api/v1/sources', { token: ADMIN, body: sourceDef({ key: 'never', endpoints: ['https://example.org/feed'] }) });
+ const h = await api('GET', '/api/v1/health', { token: READER });
+ assert.strictEqual(h.status, 200);
+ assert.ok(h.body.counts.healthy >= 1 && h.body.counts.manual === 1);
+ assert.ok(h.body.attention.some(a => a.key === 'never' && a.status === 'never_fetched'));
+ const ready = await api('GET', '/api/ready');
+ assert.strictEqual(ready.status, 200);
+ assert.strictEqual(ready.body.checks.db, true);
+});
+
+t('done', async () => { await web.close(); await svc.stop(); });
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
+<html><head>
+<script type="application/ld+json">
+[
+ { "@context": "https://schema.org", "@type": "NewsArticle", "headline": "Council approves budget", "datePublished": "2026-09-21T07:00:00+02:00", "dateModified": "2026-09-21T09:00:00+02:00", "author": [{ "@type": "Person", "name": "Lee Reporter" }], "publisher": { "@type": "Organization", "name": "Example Times" }, "description": "The council voted 7-2." },
+ { "@context": "https://schema.org", "@type": "Review", "name": "Review of Widget Pro", "itemReviewed": { "@type": "Product", "name": "Widget Pro" }, "author": { "@type": "Person", "name": "Critic" }, "reviewRating": { "@type": "Rating", "ratingValue": "3", "bestRating": "5" }, "datePublished": "2026-09-10", "reviewBody": "Solid, not spectacular." },
+ { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Why widgets", "url": "https://times.example.org/blog/why-widgets", "datePublished": "2026-09-01" }
+]
+</script>
+<script type="application/ld+json">{ this is not json }</script>
+</head></html>
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="utf-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom">
+ <title>Example Engineering Blog</title>
+ <link href="https://blog.example.com/"/>
+ <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
+ <updated>2026-09-20T18:30:02Z</updated>
+ <entry>
+ <title type="html">Release <em>2.0</em> notes</title>
+ <link rel="alternate" href="/posts/release-2"/>
+ <link rel="enclosure" href="https://blog.example.com/files/notes.pdf"/>
+ <id>tag:blog.example.com,2026:posts/release-2</id>
+ <published>2026-09-19T10:00:00Z</published>
+ <updated>2026-09-20T18:30:02Z</updated>
+ <author><name>Grace Example</name></author>
+ <category term="releases"/>
+ <summary>What changed in 2.0.</summary>
+ </entry>
+ <entry>
+ <title>Second post</title>
+ <link href="https://blog.example.com/posts/second"/>
+ <id>tag:blog.example.com,2026:posts/second</id>
+ <updated>not a date</updated>
+ <content type="html"><p>Body</p></content>
+ </entry>
+</feed>
new file mode 100644
new file mode 100644
@@ -0,0 +1,7 @@
@@ -0,0 +1,7 @@
+{
+ "deals": [
+ { "dealID": "d-1", "title": "Game One", "salePrice": "4.99", "normalPrice": "19.99", "storeID": "1", "releaseDate": 1726790400, "link": "https://deals.example.com/d/1" },
+ { "dealID": "d-2", "title": "Game Two", "salePrice": null, "normalPrice": "9.99", "storeID": "2" },
+ { "title": "No identity, skipped" }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1 @@
@@ -0,0 +1 @@
+<html><head><title>Nothing structured</title></head><body>Hello</body></html>
new file mode 100644
new file mode 100644
@@ -0,0 +1,30 @@
@@ -0,0 +1,30 @@
+<!doctype html>
+<html><head><meta charset="utf-8"><title>Widget Pro</title>
+<script type="application/ld+json">
+{
+ "@context": "https://schema.org",
+ "@graph": [
+ { "@type": "BreadcrumbList", "itemListElement": [] },
+ {
+ "@type": "Product",
+ "@id": "https://shop.example.com/p/widget-pro#product",
+ "name": "Widget Pro",
+ "sku": "WP-100",
+ "gtin13": "0012345678905",
+ "brand": { "@type": "Brand", "name": "Example Co" },
+ "description": "A <b>very</b> good widget.",
+ "image": ["https://shop.example.com/img/wp.jpg"],
+ "offers": [
+ { "@type": "Offer", "price": "19.99", "priceCurrency": "USD", "availability": "https://schema.org/InStock", "priceValidUntil": "2026-12-31", "url": "https://shop.example.com/p/widget-pro", "seller": { "@type": "Organization", "name": "Example Shop" } },
+ { "@type": "Offer", "priceCurrency": "EUR", "availability": "https://schema.org/OutOfStock" }
+ ],
+ "aggregateRating": { "@type": "AggregateRating", "ratingValue": "4.4", "reviewCount": "89" },
+ "review": [
+ { "@type": "Review", "author": { "@type": "Person", "name": "Sam" }, "reviewRating": { "@type": "Rating", "ratingValue": 5, "bestRating": 5 }, "datePublished": "2026-08-01", "reviewBody": "Long text that we do not keep." }
+ ]
+ }
+ ]
+}
+</script>
+<script type="application/ld+json">{ "@context": "https://schema.org", "@type": "Organization", "name": "Example Shop" }</script>
+</head><body>Widget</body></html>
new file mode 100644
new file mode 100644
@@ -0,0 +1,13 @@
@@ -0,0 +1,13 @@
+<?xml version="1.0"?>
+<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://purl.org/rss/1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/">
+ <channel rdf:about="https://rdf.example.net/">
+ <title>RDF Site</title>
+ <link>https://rdf.example.net/</link>
+ </channel>
+ <item rdf:about="https://rdf.example.net/a">
+ <title>RDF item A</title>
+ <link>https://rdf.example.net/a</link>
+ <dc:date>2026-09-18T08:00:00Z</dc:date>
+ <description>First.</description>
+ </item>
+</rdf:RDF>
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
+ <channel>
+ <title>Example Observatory News</title>
+ <link>https://news.example.org/</link>
+ <description>Fixture feed</description>
+ <item>
+ <title>Telescope sees first light</title>
+ <link>https://news.example.org/2026/09/first-light?utm_source=rss&id=7#top</link>
+ <guid isPermaLink="false">news-0001</guid>
+ <pubDate>Mon, 21 Sep 2026 14:30:00 +0000</pubDate>
+ <dc:creator>Ada Example</dc:creator>
+ <category>Astronomy</category>
+ <category>Instruments</category>
+ <description><![CDATA[<p>The <b>new</b> telescope & its camera saw <a href="x">first light</a>.</p><script>alert(1)</script>]]></description>
+ <enclosure url="https://news.example.org/img/first-light.jpg" type="image/jpeg" length="12345"/>
+ </item>
+ <item>
+ <title>Undated item keeps no date</title>
+ <link>https://news.example.org/2026/09/undated</link>
+ <description>Plain text description.</description>
+ </item>
+ <item>
+ <description>An item with neither guid nor link is skipped.</description>
+ </item>
+ <item>
+ <title>Duplicate guid</title>
+ <link>https://news.example.org/dup</link>
+ <guid isPermaLink="false">news-0001</guid>
+ </item>
+ </channel>
+</rss>
new file mode 100644
new file mode 100644
@@ -0,0 +1,5 @@
@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+ <sitemap><loc>https://deals.example.com/sitemap-1.xml</loc><lastmod>2026-09-01T00:00:00Z</lastmod></sitemap>
+ <sitemap><loc>https://deals.example.com/sitemap-2.xml</loc></sitemap>
+</sitemapindex>
new file mode 100644
new file mode 100644
@@ -0,0 +1,18 @@
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9">
+ <url>
+ <loc>https://deals.example.com/offers/1</loc>
+ <lastmod>2026-09-21</lastmod>
+ <changefreq>daily</changefreq>
+ <priority>0.8</priority>
+ </url>
+ <url>
+ <loc>https://deals.example.com/news/launch</loc>
+ <news:news>
+ <news:publication><news:name>Deals Example</news:name><news:language>en</news:language></news:publication>
+ <news:publication_date>2026-09-20T09:00:00Z</news:publication_date>
+ <news:title>Launch week</news:title>
+ </news:news>
+ </url>
+ <url><loc>ftp://deals.example.com/not-http</loc></url>
+</urlset>
new file mode 100644
new file mode 100644
@@ -0,0 +1,3 @@
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE lolz [<!ENTITY lol "lol"><!ENTITY lol2 "&lol;&lol;&lol;&lol;">]>
+<rss version="2.0"><channel><title>&lol2;</title><item><title>x</title><link>https://x.example/</link></item></channel></rss>
new file mode 100644
new file mode 100644
@@ -0,0 +1,141 @@
@@ -0,0 +1,141 @@
+'use strict';
+/**
+ * Shared test fixtures: a generated Network signing key, service token minting, a booted Sources
+ * service on a random port with a temp database, and stub HTTP servers standing in for the
+ * sites being ingested. Nothing here touches the internet.
+ */
+const crypto = require('crypto');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const nodeHttp = require('http');
+const { serviceAuth } = require('openvibe-contracts');
+const { load } = require('../server/config');
+const { start } = require('../server/index');
+
+const ISSUER = 'https://openvibe.network';
+const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', {
+ modulusLength: 2048,
+ privateKeyEncoding: { type: 'pkcs8', format: 'pem' },
+ publicKeyEncoding: { type: 'spki', format: 'pem' },
+});
+
+const silent = { log() {}, warn() {}, error(...a) { if (process.env.DEBUG) console.error(...a); } };
+
+function serviceToken(slug, cap, { aud = 'openvibe.sources', exp = Math.floor(Date.now() / 1000) + 300, key = privateKey } = {}) {
+ const now = Math.floor(Date.now() / 1000);
+ return serviceAuth.signServiceToken({
+ iss: ISSUER, sub: `svc:${slug}`, actor_type: 'service', aud: [aud], cap, ns: [], iat: now, exp,
+ jti: `tok_${crypto.randomBytes(8).toString('hex')}`,
+ }, key);
+}
+
+const made = [];
+process.on('exit', () => { for (const d of made) fs.rmSync(d, { recursive: true, force: true }); });
+
+function tmpDir() {
+ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-sources-test-'));
+ made.push(d);
+ return d;
+}
+
+/** Boot Sources; the scheduler is off unless worker: 'on'. Loopback is allowlisted for the stubs. */
+async function boot({ env = {}, worker = 'off', lookupImpl, tokenClient } = {}) {
+ const dir = tmpDir();
+ const config = load({
+ NODE_ENV: 'test',
+ PORT: '0',
+ SOURCES_DB_PATH: path.join(dir, 'sources.db'),
+ OV_NETWORK_PUBLIC_KEY: publicKey,
+ SOURCES_WORKER: worker,
+ SOURCES_ALLOW_PRIVATE_HOSTS: '127.0.0.1',
+ SOURCES_HOST_MIN_INTERVAL_MS: '0',
+ SOURCES_FETCH_TIMEOUT_MS: '1500',
+ SOURCES_TICK_MS: '50',
+ ...env,
+ });
+ const h = await start({ config, log: silent, lookupImpl, tokenClient });
+ const base = `http://127.0.0.1:${h.server.address().port}`;
+ return { ...h, base, dir, async stop() { await h.close(); } };
+}
+
+async function request(base, method, p, { token, body, headers = {} } = {}) {
+ const h = { ...headers };
+ if (token) h.Authorization = `Bearer ${token}`;
+ if (body !== undefined) h['Content-Type'] = 'application/json';
+ const res = await fetch(base + p, { method, headers: h, body: body !== undefined ? JSON.stringify(body) : undefined });
+ const text = await res.text();
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { json = null; }
+ return { status: res.status, body: json, text, headers: res.headers };
+}
+
+/**
+ * A stub website. routes: { '/path': (req, res, ctx) => void | { status, headers, body, delayMs } }.
+ * Every request is recorded with its headers and arrival time.
+ */
+async function site(routes = {}) {
+ const requests = [];
+ const server = nodeHttp.createServer(async (req, res) => {
+ const url = new URL(req.url, 'http://stub');
+ requests.push({ path: url.pathname, search: url.search, headers: req.headers, at: Date.now() });
+ const handler = routes[url.pathname] || routes['*'];
+ if (!handler) { res.statusCode = 404; res.end('not found'); return; }
+ const out = await handler(req, res, { url, count: requests.filter(r => r.path === url.pathname).length });
+ if (!out || res.writableEnded) return;
+ if (out.delayMs) await new Promise(r => setTimeout(r, out.delayMs));
+ if (res.destroyed) return;
+ res.writeHead(out.status || 200, out.headers || {});
+ res.end(out.body === undefined ? '' : out.body);
+ });
+ await new Promise(r => server.listen(0, '127.0.0.1', r));
+ const origin = `http://127.0.0.1:${server.address().port}`;
+ return {
+ origin,
+ requests,
+ hits: (p) => requests.filter(r => r.path === p),
+ routes,
+ close: () => new Promise(r => { server.closeAllConnections?.(); server.close(() => r()); }),
+ };
+}
+
+/** A minimal valid source definition pointing at a stub. */
+function sourceDef(overrides = {}) {
+ return {
+ key: 'src-' + crypto.randomBytes(4).toString('hex'),
+ name: 'Stub source',
+ type: 'rss',
+ category: 'news',
+ endpoints: [],
+ robots_note: 'robots.txt checked by the fetcher on every run',
+ terms_note: 'test fixture',
+ min_interval_ms: 0,
+ poll_interval_sec: 3600,
+ enabled: true,
+ ...overrides,
+ };
+}
+
+function rss(items) {
+ return `<?xml version="1.0"?><rss version="2.0"><channel><title>Stub</title>${items.map(i =>
+ `<item><title>${i.title}</title><link>${i.link}</link><guid isPermaLink="false">${i.guid}</guid>${i.date ? `<pubDate>${i.date}</pubDate>` : ''}<description>${i.description || ''}</description></item>`).join('')}</channel></rss>`;
+}
+
+const fixture = (name) => fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');
+const sleep = (ms) => new Promise(r => setTimeout(r, ms));
+
+function suite(name) {
+ const tests = [];
+ const t = (n, fn) => tests.push([n, fn]);
+ t.run = async () => {
+ let failed = 0;
+ for (const [n, fn] of tests) {
+ try { await fn(); console.log(` ok ${n}`); } catch (err) { failed++; console.log(` FAIL ${n}\n${err.stack}`); }
+ }
+ console.log(`${name}: ${tests.length - failed}/${tests.length} passed`);
+ if (failed) process.exit(1);
+ };
+ return t;
+}
+
+module.exports = { ISSUER, privateKey, publicKey, silent, serviceToken, tmpDir, boot, request, site, sourceDef, rss, fixture, sleep, suite };
new file mode 100644
new file mode 100644
@@ -0,0 +1,337 @@
@@ -0,0 +1,337 @@
+'use strict';
+/**
+ * Ingestion against stub sites: explicit run states, conditional GET, failure never fabricates or
+ * modifies items, staleness, rate limits, one run in flight, credentials by variable name.
+ */
+const assert = require('assert');
+const { boot, site, sourceDef, rss, fixture, sleep, suite } = require('./helpers');
+
+const t = suite('ingest');
+let svc;
+let web;
+let feedBody;
+let feedStatus;
+let feedEtag;
+
+const FEED_V1 = rss([
+ { guid: 'a-1', title: 'Alpha', link: 'https://news.example.org/a', date: 'Mon, 21 Sep 2026 10:00:00 +0000', description: 'first' },
+ { guid: 'b-1', title: 'Beta', link: 'https://news.example.org/b' },
+]);
+
+function create(def) {
+ svc.registry.create(def, 'test');
+ return def.key;
+}
+const run = (key, trigger = 'manual') => svc.ingest.run(key, { trigger });
+const itemsOf = (key) => svc.db.prepare('SELECT * FROM items WHERE source_key = ? ORDER BY identity').all(key);
+const runsOf = (key) => svc.db.prepare('SELECT * FROM fetch_runs WHERE source_key = ? ORDER BY rid').all(key);
+const events = (type) => svc.outbox.all().filter(e => e.event_type === type);
+const reset = (key) => svc.db.prepare('UPDATE sources SET last_request_at = NULL, not_before = 0 WHERE key = ?').run(key);
+
+t('boot', async () => {
+ svc = await boot();
+ feedBody = FEED_V1;
+ feedStatus = 200;
+ feedEtag = '"v1"';
+ web = await site({
+ '/robots.txt': () => ({ body: 'User-agent: *\nDisallow: /private/\n\nUser-agent: OpenVibeSources\nDisallow: /blocked-for-us\nAllow: /\n' }),
+ '/feed.xml': (req) => {
+ if (feedStatus !== 200) return { status: feedStatus, body: 'nope' };
+ if (feedEtag && req.headers['if-none-match'] === feedEtag) return { status: 304 };
+ return { headers: { 'Content-Type': 'application/rss+xml; charset=utf-8', ETag: feedEtag }, body: feedBody };
+ },
+ });
+});
+
+t('a good fetch creates items with full provenance and an ok run', async () => {
+ const key = create(sourceDef({ key: 'news-a', endpoints: [`${web.origin}/feed.xml`], license_note: 'headlines only', terms_note: 'fixture terms' }));
+ const out = await run(key);
+ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok']);
+ const items = itemsOf(key);
+ assert.strictEqual(items.length, 2);
+ const a = items[0];
+ assert.strictEqual(a.identity, 'a-1');
+ assert.strictEqual(a.canonical_url, 'https://news.example.org/a');
+ assert.strictEqual(a.revision, 1);
+ assert.match(a.content_hash, /^[0-9a-f]{64}$/);
+ assert.match(a.raw_body_hash, /^[0-9a-f]{64}$/);
+ assert.strictEqual(a.parser_version, 'feed@1');
+ assert.strictEqual(a.license_note, 'headlines only');
+ assert.strictEqual(a.terms_note, 'fixture terms');
+ assert.ok(a.retrieved_at > 0 && a.first_seen_at === a.retrieved_at);
+ assert.strictEqual(a.published_at, '2026-09-21T10:00:00.000Z');
+ assert.strictEqual(items[1].published_at, null);
+ const [r] = runsOf(key);
+ assert.deepStrictEqual([r.state, r.http_status, r.items_created, r.items_seen], ['ok', 200, 2, 2]);
+ assert.strictEqual(r.raw_body_hash, a.raw_body_hash);
+ assert.strictEqual(events('sources.item.created').filter(e => e.payload.source_key === key).length, 2);
+ // the fetcher identified itself and asked robots.txt first
+ const feedReq = web.hits('/feed.xml')[0];
+ assert.match(feedReq.headers['user-agent'], /^OpenVibeSources\//);
+ assert.ok(web.hits('/robots.txt')[0].at <= feedReq.at);
+});
+
+t('conditional GET: the stored ETag is sent and a 304 is not_modified with nothing changed', async () => {
+ reset('news-a');
+ const before = itemsOf('news-a');
+ const out = await run('news-a');
+ assert.deepStrictEqual(out.runs.map(r => r.state), ['not_modified']);
+ const last = web.hits('/feed.xml').at(-1);
+ assert.strictEqual(last.headers['if-none-match'], '"v1"');
+ assert.deepStrictEqual(itemsOf('news-a').map(i => [i.revision, i.content_hash, i.retrieved_at]), before.map(i => [i.revision, i.content_hash, i.retrieved_at]));
+});
+
+t('Last-Modified is used when there is no ETag', async () => {
+ const lm = 'Mon, 21 Sep 2026 10:00:00 GMT';
+ const s = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/f': (req) => (req.headers['if-modified-since'] === lm ? { status: 304 } : { headers: { 'Last-Modified': lm }, body: FEED_V1 }),
+ });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/f`] }));
+ assert.strictEqual((await run(key)).runs[0].state, 'ok');
+ reset(key);
+ assert.strictEqual((await run(key)).runs[0].state, 'not_modified');
+ } finally { await s.close(); }
+});
+
+t('a changed item is a new revision; the old one stays in item_revisions', async () => {
+ reset('news-a');
+ feedEtag = '"v2"';
+ feedBody = rss([
+ { guid: 'a-1', title: 'Alpha (corrected)', link: 'https://news.example.org/a', date: 'Mon, 21 Sep 2026 10:00:00 +0000', description: 'first, corrected' },
+ { guid: 'b-1', title: 'Beta', link: 'https://news.example.org/b' },
+ ]);
+ const out = await run('news-a');
+ assert.strictEqual(out.runs[0].state, 'ok');
+ assert.deepStrictEqual([out.runs[0].items_updated, out.runs[0].items_unchanged], [1, 1]);
+ const a = itemsOf('news-a')[0];
+ assert.strictEqual(a.revision, 2);
+ assert.strictEqual(a.title, 'Alpha (corrected)');
+ const revs = svc.db.prepare('SELECT revision, snapshot FROM item_revisions WHERE item_id = ? ORDER BY revision').all(a.id);
+ assert.deepStrictEqual(revs.map(r => [r.revision, JSON.parse(r.snapshot).title]), [[1, 'Alpha'], [2, 'Alpha (corrected)']]);
+ const upd = events('sources.item.updated').find(e => e.payload.item_id === a.id);
+ assert.ok(upd && upd.payload.previous_content_hash && upd.payload.revision === 2);
+});
+
+t('failure never fabricates: http errors, timeouts, parse errors and oversize bodies leave items exactly as they were', async () => {
+ const snapshot = () => JSON.stringify(itemsOf('news-a'));
+ const before = snapshot();
+ const count = svc.db.prepare('SELECT COUNT(*) AS n FROM items').get().n;
+
+ feedStatus = 500;
+ reset('news-a');
+ let out = await run('news-a');
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].http_status, out.runs[0].error_code], ['http_error', 500, 'http_500']);
+
+ feedStatus = 200;
+ feedEtag = '"v3"';
+ feedBody = '<rss><channel><item><title>Broken';
+ reset('news-a');
+ out = await run('news-a');
+ assert.strictEqual(out.runs[0].state, 'parse_error');
+ assert.match(out.runs[0].raw_body_hash, /^[0-9a-f]{64}$/, 'the unreadable body is identified by its hash');
+
+ feedBody = '<html><body>Service temporarily replaced by a login page</body></html>';
+ reset('news-a');
+ out = await run('news-a');
+ assert.strictEqual(out.runs[0].state, 'parse_error');
+
+ assert.strictEqual(snapshot(), before, 'no item created, changed, re-dated or re-hashed');
+ assert.strictEqual(svc.db.prepare('SELECT COUNT(*) AS n FROM items').get().n, count);
+
+ // after a failed parse the validator was not stored: the next good body is fetched in full
+ feedBody = FEED_V1;
+ reset('news-a');
+ out = await run('news-a');
+ assert.strictEqual(out.runs[0].state, 'ok');
+ assert.notStrictEqual(web.hits('/feed.xml').at(-1).headers['if-none-match'], '"v3"');
+
+ const failed = events('sources.fetch.failed').filter(e => e.payload.source_key === 'news-a');
+ assert.deepStrictEqual(failed.map(e => e.payload.state), ['http_error', 'parse_error', 'parse_error']);
+ assert.deepStrictEqual(failed.map(e => e.payload.consecutive_failures), [1, 2, 3]);
+ assert.strictEqual(svc.registry.get('news-a').consecutive_failures, 0, 'reset by the good run');
+});
+
+t('timeouts and oversize bodies are explicit states', async () => {
+ const s = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/slow': () => ({ delayMs: 3000, body: FEED_V1 }),
+ '/huge': () => ({ body: 'x'.repeat(4096) }),
+ });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/slow`] }));
+ const out = await run(key);
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['timeout', 'timeout']);
+ assert.strictEqual(itemsOf(key).length, 0);
+
+ const small = await boot({ env: { SOURCES_MAX_BYTES: '1024' } });
+ try {
+ small.registry.create(sourceDef({ key: 'huge', endpoints: [`${s.origin}/huge`] }), 'test');
+ const r = await small.ingest.run('huge', { trigger: 'manual' });
+ assert.deepStrictEqual([r.runs[0].state, r.runs[0].error_code], ['http_error', 'too_large']);
+ } finally { await small.stop(); }
+ } finally { await s.close(); }
+});
+
+t('staleness is computed from the last success and exposed in health', async () => {
+ const key = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], stale_after_sec: 1 }));
+ let h = svc.registry.health(svc.registry.get(key));
+ assert.deepStrictEqual([h.status, h.stale], ['never_fetched', true]);
+ feedEtag = null;
+ await run(key);
+ h = svc.registry.health(svc.registry.get(key));
+ assert.deepStrictEqual([h.status, h.stale], ['healthy', false]);
+ await sleep(1100);
+ h = svc.registry.health(svc.registry.get(key));
+ assert.deepStrictEqual([h.status, h.stale], ['stale', true]);
+ feedStatus = 503;
+ reset(key);
+ await run(key);
+ h = svc.registry.health(svc.registry.get(key));
+ assert.deepStrictEqual([h.status, h.stale, h.consecutive_failures], ['failing', true, 1]);
+ feedStatus = 200;
+});
+
+t('rate limit: requests to a source are spaced by min_interval_ms; an early trigger is rate_limited without a request', async () => {
+ const s = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/one': () => ({ body: FEED_V1 }),
+ '/two': () => ({ body: FEED_V1 }),
+ });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/one`, `${s.origin}/two`], min_interval_ms: 400 }));
+ const out = await run(key);
+ assert.deepStrictEqual(out.runs.map(r => r.state), ['ok', 'ok']);
+ const gap = s.hits('/two')[0].at - s.hits('/one')[0].at;
+ assert.ok(gap >= 380, `requests ${gap} ms apart`);
+ const again = await run(key);
+ assert.deepStrictEqual([again.runs[0].state, again.runs[0].error_code], ['rate_limited', 'local_rate_limit']);
+ assert.strictEqual(s.hits('/one').length, 1, 'nothing was requested');
+ assert.ok(!events('sources.fetch.failed').some(e => e.payload.source_key === key), 'our own limiter is not a source failure');
+ } finally { await s.close(); }
+});
+
+t('a 429 is rate_limited, stops the run and honours Retry-After', async () => {
+ const s = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/a': () => ({ status: 429, headers: { 'Retry-After': '120' } }),
+ '/b': () => ({ body: FEED_V1 }),
+ });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/a`, `${s.origin}/b`] }));
+ const out = await run(key);
+ assert.deepStrictEqual(out.runs.map(r => [r.state, r.error_code]), [['rate_limited', 'upstream_429']]);
+ assert.strictEqual(s.hits('/b').length, 0);
+ const row = svc.registry.get(key);
+ assert.ok(row.not_before - Date.now() > 100 * 1000, 'Retry-After respected');
+ const early = await run(key);
+ assert.strictEqual(early.runs[0].state, 'rate_limited');
+ assert.strictEqual(s.hits('/a').length, 1);
+ } finally { await s.close(); }
+});
+
+t('one run per source in flight', async () => {
+ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/slow': () => ({ delayMs: 300, body: FEED_V1 }) });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/slow`] }));
+ const first = run(key);
+ await sleep(50);
+ const second = await run(key);
+ assert.deepStrictEqual(second, { busy: true });
+ assert.strictEqual((await first).runs[0].state, 'ok');
+ assert.strictEqual(s.hits('/slow').length, 1);
+ } finally { await s.close(); }
+});
+
+t('disabled sources and missing credentials are recorded as disabled, nothing is requested', async () => {
+ const key = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], enabled: false }));
+ const n = web.requests.length;
+ const out = await run(key);
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['disabled', 'source_disabled']);
+ const key2 = create(sourceDef({ endpoints: [`${web.origin}/feed.xml`], auth: { mode: 'bearer', env: 'SOURCES_CRED_NOT_SET' } }));
+ const out2 = await run(key2);
+ assert.deepStrictEqual([out2.runs[0].state, out2.runs[0].error_code], ['disabled', 'credential_missing']);
+ assert.match(out2.runs[0].detail, /SOURCES_CRED_NOT_SET/);
+ assert.strictEqual(web.requests.length, n);
+});
+
+t('a credential is read by name, sent only to its origin, and never stored', async () => {
+ const other = await site({ '/robots.txt': () => ({ status: 404 }), '/landing': () => ({ body: FEED_V1 }) });
+ const s = await site({
+ '/robots.txt': () => ({ status: 404 }),
+ '/api': () => ({ status: 302, headers: { Location: `${other.origin}/landing` } }),
+ });
+ const secret = 'sk_test_' + 'q'.repeat(24);
+ const svc2 = await boot({ env: { SOURCES_CRED_EXAMPLE: secret } });
+ try {
+ svc2.registry.create(sourceDef({ key: 'cred', endpoints: [`${s.origin}/api`], auth: { mode: 'header', env: 'SOURCES_CRED_EXAMPLE', header: 'X-Api-Key' } }), 'test');
+ const out = await svc2.ingest.run('cred', { trigger: 'manual' });
+ assert.strictEqual(out.runs[0].state, 'ok');
+ assert.strictEqual(s.hits('/api')[0].headers['x-api-key'], secret);
+ assert.strictEqual(other.hits('/landing')[0].headers['x-api-key'], undefined, 'not forwarded across origins');
+ const dump = JSON.stringify(svc2.db.prepare('SELECT * FROM sources').all()) + JSON.stringify(svc2.db.prepare('SELECT * FROM fetch_runs').all())
+ + JSON.stringify(svc2.db.prepare('SELECT * FROM items').all()) + JSON.stringify(svc2.outbox.all());
+ assert.ok(!dump.includes(secret), 'the secret value is nowhere in the database or events');
+ assert.ok(dump.includes('SOURCES_CRED_EXAMPLE'), 'the variable name is recorded');
+ } finally { await svc2.stop(); await s.close(); await other.close(); }
+});
+
+t('removal is explicit and sticky; the next fetch does not bring the item back', async () => {
+ const key = 'news-a';
+ const a = itemsOf(key).find(i => i.identity === 'a-1');
+ const source = svc.registry.fromRow(svc.registry.get(key));
+ const removed = svc.items.remove(source, a.id, 'publisher takedown request', 'svc:network');
+ assert.ok(removed.removed && /takedown/.test(removed.removed.reason));
+ reset(key);
+ feedEtag = '"v9"';
+ feedBody = FEED_V1;
+ await run(key);
+ const again = svc.items.get(a.id);
+ assert.ok(again.removed, 'still removed');
+ assert.ok(events('sources.item.removed').some(e => e.payload.item_id === a.id));
+});
+
+t('JSON-LD and sitemap sources ingest end to end', async () => {
+ const s = await site({
+ '/robots.txt': () => ({ body: 'User-agent: *\nAllow: /\n' }),
+ '/p/widget-pro': () => ({ headers: { 'Content-Type': 'text/html; charset=utf-8' }, body: fixture('product.html') }),
+ '/sitemap.xml': () => ({ headers: { 'Content-Type': 'application/xml' }, body: fixture('sitemap.xml') }),
+ });
+ try {
+ const p = create(sourceDef({ type: 'jsonld', category: 'deals', endpoints: [`${s.origin}/p/widget-pro`] }));
+ assert.strictEqual((await run(p)).runs[0].state, 'ok');
+ const [prod] = itemsOf(p);
+ assert.strictEqual(prod.kind, 'product');
+ assert.strictEqual(JSON.parse(prod.fields).offers[0].price, '19.99');
+ assert.strictEqual(prod.parser_version, 'jsonld@1');
+ const sm = create(sourceDef({ type: 'sitemap', category: 'deals', endpoints: [`${s.origin}/sitemap.xml`] }));
+ const out = await run(sm);
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].items_created, out.runs[0].items_skipped], ['ok', 2, 1]);
+ } finally { await s.close(); }
+});
+
+t('sources with search_visibility send staff-only, noindex index documents through the outbox', async () => {
+ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: FEED_V1 }) });
+ try {
+ const key = create(sourceDef({ endpoints: [`${s.origin}/f`], search_visibility: 'members' }));
+ await run(key);
+ const docs = events('sources.index_document.upserted').filter(e => e.payload.facets.source === key);
+ assert.strictEqual(docs.length, 2);
+ const d = docs[0].payload;
+ assert.deepStrictEqual([d.owner, d.type, d.visibility, d.indexability.decision], ['sources', 'item', 'members', 'noindex']);
+ assert.deepStrictEqual(d.acl.groups, ['role:admin', 'role:global_mod']);
+ assert.deepStrictEqual(docs[0].subject, { type: 'item', id: d.id, revision: d.revision });
+ assert.strictEqual(docs[0].visibility, 'internal');
+ const item = itemsOf(key)[0];
+ svc.items.remove(svc.registry.fromRow(svc.registry.get(key)), item.id, 'test', null);
+ const del = events('sources.index_document.deleted').find(e => e.payload.id === item.id);
+ assert.deepStrictEqual(del.payload, { type: 'item', id: item.id, revision: 2 });
+ // a source without search_visibility sends none
+ assert.ok(!events('sources.index_document.upserted').some(e => e.payload.facets.source === 'news-a'));
+ } finally { await s.close(); }
+});
+
+t('done', async () => { await web.close(); await svc.stop(); });
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,80 @@
@@ -0,0 +1,80 @@
+'use strict';
+/**
+ * Proposals for OpenVibe.Contracts are valid against the contracts' own schemas; what the API
+ * returns matches the proposed sources.source@1 / sources.item@1; the seeds are honest.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const Ajv2020 = require('ajv/dist/2020');
+const addFormats = require('ajv-formats');
+const contracts = require('openvibe-contracts');
+const { CAPS } = require('../server/auth');
+const { validateSource, CATEGORIES } = require('../server/registry');
+const { seed } = require('../scripts/seed');
+const { boot, request, serviceToken, site, sourceDef, rss, suite } = require('./helpers');
+
+const t = suite('proposals');
+const DOCS = path.join(__dirname, '..', 'docs');
+const ajv = new Ajv2020({ allErrors: true, strict: false });
+addFormats(ajv);
+const schema = (name) => ajv.compile(JSON.parse(fs.readFileSync(path.join(DOCS, 'contracts-proposal', 'contracts', 'sources', name), 'utf8')));
+const sourceSchema = schema('source.v1.json');
+const itemSchema = schema('item.v1.json');
+
+t('capability proposals are valid capabilities.capability@1 and cover every enforced id', () => {
+ const dir = path.join(DOCS, 'capabilities-proposal');
+ const ids = fs.readdirSync(dir).filter(f => f.endsWith('.json')).map((f) => {
+ const m = JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8'));
+ const v = contracts.validate('capabilities.capability@1', m);
+ assert.ok(v.valid, `${f}: ${JSON.stringify(v.errors)}`);
+ assert.strictEqual(`${m.id}.json`, f);
+ return m.id;
+ });
+ assert.deepStrictEqual(ids.sort(), Object.values(CAPS).sort());
+});
+
+t('the service manifest proposal is a valid registry.service-manifest@1', () => {
+ const m = JSON.parse(fs.readFileSync(path.join(DOCS, 'service-manifest-proposal.json'), 'utf8'));
+ const v = contracts.validate('registry.service-manifest@1', m);
+ assert.ok(v.valid, JSON.stringify(v.errors));
+ assert.deepStrictEqual([...m.capabilities].sort(), Object.values(CAPS).sort());
+});
+
+t('seeds: one real source per category, all disabled, all valid, notes recorded', () => {
+ const seeds = require('../seeds/sources.json').sources;
+ assert.deepStrictEqual(seeds.map(s => s.category).sort(), [...CATEGORIES].sort());
+ for (const s of seeds) {
+ const rec = validateSource(s);
+ assert.strictEqual(rec.enabled, false, s.key);
+ assert.ok(rec.terms_note, `${s.key} has a terms note`);
+ if (rec.type !== 'manual') assert.ok(rec.robots_note, `${s.key} has a robots note`);
+ assert.strictEqual(rec.default_indexability, 'noindex');
+ for (const ep of rec.endpoints) assert.ok(ep.url.startsWith('https://'), ep.url);
+ assert.ok(sourceSchema(rec), JSON.stringify(sourceSchema.errors));
+ }
+ const { openDb } = require('../server/db');
+ const db = openDb(':memory:');
+ assert.strictEqual(seed(db).created.length, seeds.length);
+ assert.deepStrictEqual(seed(db).created, [], 'never overwrites');
+ assert.strictEqual(db.prepare('SELECT COUNT(*) AS n FROM sources WHERE enabled = 1').get().n, 0);
+ db.close();
+});
+
+t('API views match sources.source@1 and sources.item@1', async () => {
+ const svc = await boot();
+ const web = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: rss([{ guid: 'p1', title: 'P', link: 'https://example.org/p' }]) }) });
+ try {
+ const admin = serviceToken('network', ['sources.*']);
+ await request(svc.base, 'POST', '/api/v1/sources', { token: admin, body: sourceDef({ key: 'proposal-check', endpoints: [`${web.origin}/f`] }) });
+ await request(svc.base, 'POST', '/api/v1/sources/proposal-check/fetch', { token: admin });
+ const src = await request(svc.base, 'GET', '/api/v1/sources/proposal-check', { token: admin });
+ assert.ok(sourceSchema(src.body.source), JSON.stringify(sourceSchema.errors));
+ const items = await request(svc.base, 'GET', '/api/v1/items', { token: admin });
+ for (const it of items.body.items) assert.ok(itemSchema(it), JSON.stringify(itemSchema.errors));
+ const one = await request(svc.base, 'GET', `/api/v1/items/${items.body.items[0].id}?revisions=1`, { token: admin });
+ assert.ok(itemSchema(one.body.item), JSON.stringify(itemSchema.errors));
+ } finally { await svc.stop(); await web.close(); }
+});
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,132 @@
@@ -0,0 +1,132 @@
+'use strict';
+/** robots.txt is respected (rules, unreachable = deny, redirects) and internal addresses are never fetched. */
+const assert = require('assert');
+const { parseRobots, isAllowed } = require('../server/robots');
+const { isPublicAddress } = require('../server/net/guard');
+const { boot, site, sourceDef, rss, suite } = require('./helpers');
+
+const t = suite('robots-ssrf');
+const FEED = rss([{ guid: 'x', title: 'X', link: 'https://example.org/x' }]);
+
+t('robots rules: our group over *, longest match, Allow wins ties, * and $', () => {
+ const txt = [
+ 'User-agent: *', 'Disallow: /', '',
+ 'User-agent: OpenVibeSources', 'User-agent: other-bot', 'Disallow: /private', 'Allow: /private/press', 'Disallow: /*.pdf$', 'Crawl-delay: 2', '',
+ 'User-agent: openvibesources', 'Disallow: /tmp/',
+ ].join('\n');
+ const p = parseRobots(txt, 'OpenVibeSources');
+ assert.strictEqual(p.crawlDelaySec, 2);
+ assert.strictEqual(isAllowed(p, '/'), true, 'the * group does not apply when ours exists');
+ assert.strictEqual(isAllowed(p, '/private/x'), false);
+ assert.strictEqual(isAllowed(p, '/private/press/2026'), true);
+ assert.strictEqual(isAllowed(p, '/docs/a.pdf'), false);
+ assert.strictEqual(isAllowed(p, '/docs/a.pdf?x=1'), true);
+ assert.strictEqual(isAllowed(p, '/tmp/file'), false, 'groups for the same agent merge');
+ const star = parseRobots('User-agent: *\nDisallow: /a\nAllow: /a\n', 'OpenVibeSources');
+ assert.strictEqual(isAllowed(star, '/a/b'), true, 'Allow wins a tie');
+ const empty = parseRobots('User-agent: *\nDisallow:\n', 'OpenVibeSources');
+ assert.strictEqual(isAllowed(empty, '/anything'), true);
+ const prefix = parseRobots('User-agent: OpenVibe\nDisallow: /\n', 'OpenVibeSources');
+ assert.strictEqual(isAllowed(prefix, '/x'), true, 'a different token (even a prefix of ours) does not apply');
+});
+
+t('address policy', () => {
+ for (const ip of ['127.0.0.1', '10.1.2.3', '172.16.0.1', '192.168.1.1', '169.254.169.254', '100.64.0.1', '0.0.0.0', '::1', 'fe80::1', 'fd00::1', '::ffff:127.0.0.1', '::ffff:10.0.0.1', '64:ff9b::a00:1', '2002:c0a8:101::1', '224.0.0.1']) {
+ assert.strictEqual(isPublicAddress(ip), false, ip);
+ }
+ for (const ip of ['93.184.216.34', '1.1.1.1', '2606:4700:4700::1111']) assert.strictEqual(isPublicAddress(ip), true, ip);
+});
+
+let svc;
+let web;
+
+t('boot', async () => {
+ svc = await boot();
+ web = await site({
+ '/robots.txt': () => ({ body: 'User-agent: *\nDisallow: /private/\n' }),
+ '/private/feed.xml': () => ({ body: FEED }),
+ '/public/feed.xml': () => ({ body: FEED }),
+ '/public/moved': () => ({ status: 301, headers: { Location: '/private/feed.xml' } }),
+ });
+});
+
+t('a disallowed endpoint is robots_denied and never requested', async () => {
+ svc.registry.create(sourceDef({ key: 'denied', endpoints: [`${web.origin}/private/feed.xml`, `${web.origin}/public/feed.xml`] }), 'test');
+ const out = await svc.ingest.run('denied', { trigger: 'manual' });
+ assert.deepStrictEqual(out.runs.map(r => r.state), ['robots_denied', 'ok']);
+ assert.strictEqual(web.hits('/private/feed.xml').length, 0);
+ assert.strictEqual(web.hits('/robots.txt').length, 1, 'robots.txt is cached per origin');
+ assert.ok(svc.outbox.all().some(e => e.event_type === 'sources.fetch.failed' && e.payload.state === 'robots_denied'));
+});
+
+t('a redirect into a disallowed path is refused before it is followed', async () => {
+ svc.registry.create(sourceDef({ key: 'hop', endpoints: [`${web.origin}/public/moved`] }), 'test');
+ const out = await svc.ingest.run('hop', { trigger: 'manual' });
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['robots_denied', 'hop_refused']);
+ assert.strictEqual(web.hits('/private/feed.xml').length, 0);
+});
+
+t('robots.txt 5xx or unreachable means complete disallow; 404 means allowed', async () => {
+ const down = await site({ '/robots.txt': () => ({ status: 503 }), '/feed.xml': () => ({ body: FEED }) });
+ const none = await site({ '/robots.txt': () => ({ status: 404 }), '/feed.xml': () => ({ body: FEED }) });
+ try {
+ svc.registry.create(sourceDef({ key: 'robots-down', endpoints: [`${down.origin}/feed.xml`] }), 'test');
+ svc.registry.create(sourceDef({ key: 'robots-none', endpoints: [`${none.origin}/feed.xml`] }), 'test');
+ assert.strictEqual((await svc.ingest.run('robots-down', { trigger: 'manual' })).runs[0].state, 'robots_denied');
+ assert.strictEqual(down.hits('/feed.xml').length, 0);
+ assert.strictEqual((await svc.ingest.run('robots-none', { trigger: 'manual' })).runs[0].state, 'ok');
+ } finally { await down.close(); await none.close(); }
+});
+
+t('Crawl-delay spaces requests to the host', async () => {
+ const s = await site({ '/robots.txt': () => ({ body: 'User-agent: *\nCrawl-delay: 1\n' }), '/a': () => ({ body: FEED }), '/b': () => ({ body: FEED }) });
+ try {
+ svc.registry.create(sourceDef({ key: 'crawl-delay', endpoints: [`${s.origin}/a`, `${s.origin}/b`] }), 'test');
+ await svc.ingest.run('crawl-delay', { trigger: 'manual' });
+ const gap = s.hits('/b')[0].at - s.hits('/a')[0].at;
+ assert.ok(gap >= 950, `gap ${gap} ms`);
+ } finally { await s.close(); }
+});
+
+t('loopback and private addresses are refused unless explicitly allowlisted', async () => {
+ const strict = await boot({ env: { SOURCES_ALLOW_PRIVATE_HOSTS: '' } });
+ try {
+ strict.registry.create(sourceDef({ key: 'loop', endpoints: [`${web.origin}/public/feed.xml`] }), 'test');
+ const out = await strict.ingest.run('loop', { trigger: 'manual' });
+ assert.strictEqual(out.runs[0].state, 'http_error');
+ assert.ok(['address_refused', 'port_refused'].includes(out.runs[0].error_code), out.runs[0].error_code);
+ const n = web.requests.length;
+ strict.registry.create(sourceDef({ key: 'meta', endpoints: ['http://169.254.169.254/latest/meta-data/'] }), 'test');
+ const meta = await strict.ingest.run('meta', { trigger: 'manual' });
+ assert.deepStrictEqual([meta.runs[0].state, meta.runs[0].error_code], ['http_error', 'address_refused']);
+ assert.strictEqual(web.requests.length, n);
+ } finally { await strict.stop(); }
+});
+
+t('a public-looking name that resolves to a private address is refused at connect time', async () => {
+ // DNS stub: "feeds.example.com" resolves to 127.0.0.1 (a rebinding-style answer)
+ const lookupImpl = (host, opts, cb) => cb(null, [{ address: '127.0.0.1', family: 4 }]);
+ const strict = await boot({ env: { SOURCES_ALLOW_PRIVATE_HOSTS: '', SOURCES_ALLOWED_PORTS: `80,443,${new URL(web.origin).port}` }, lookupImpl });
+ try {
+ const port = new URL(web.origin).port;
+ strict.registry.create(sourceDef({ key: 'rebind', endpoints: [`http://feeds.example.com:${port}/public/feed.xml`] }), 'test');
+ const n = web.requests.length;
+ const out = await strict.ingest.run('rebind', { trigger: 'manual' });
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['http_error', 'address_refused']);
+ assert.strictEqual(web.requests.length, n, 'no connection was made');
+ assert.strictEqual(strict.db.prepare('SELECT COUNT(*) AS n FROM items').get().n, 0);
+ } finally { await strict.stop(); }
+});
+
+t('redirects to an internal address are refused', async () => {
+ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/r': () => ({ status: 302, headers: { Location: 'http://10.0.0.5/admin' } }) });
+ try {
+ svc.registry.create(sourceDef({ key: 'redir-internal', endpoints: [`${s.origin}/r`] }), 'test');
+ const out = await svc.ingest.run('redir-internal', { trigger: 'manual' });
+ assert.deepStrictEqual([out.runs[0].state, out.runs[0].error_code], ['http_error', 'address_refused']);
+ } finally { await s.close(); }
+});
+
+t('done', async () => { await web.close(); await svc.stop(); });
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,56 @@
@@ -0,0 +1,56 @@
+#!/usr/bin/env node
+/**
+ * Runs every test in test/ — the files named *.test.js — each in its own process, and fails
+ * if any of them fails. They use temp SQLite databases, generated RSA keys and stub subscribers
+ * on random ports; none of them needs the network or a running OpenVibe.Network.
+ *
+ * npm test # everything
+ * npm test -- publish sse # only files whose name contains one of the words
+ */
+'use strict';
+const { spawn } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+
+const filters = process.argv.slice(2).filter((a) => !a.startsWith('-'));
+const files = fs.readdirSync(__dirname)
+ .filter((f) => f.endsWith('.test.js'))
+ .filter((f) => !filters.length || filters.some((w) => f.includes(w)))
+ .sort();
+const TIMEOUT_MS = 60000;
+
+function runOne(file) {
+ return new Promise((resolve) => {
+ const started = Date.now();
+ const child = spawn(process.execPath, [path.join(__dirname, file)], {
+ cwd: path.join(__dirname, '..'),
+ env: { ...process.env, NODE_ENV: 'test' },
+ stdio: ['ignore', 'pipe', 'pipe'],
+ });
+ let output = '';
+ child.stdout.on('data', (c) => { output += c; });
+ child.stderr.on('data', (c) => { output += c; });
+ const timer = setTimeout(() => { output += `\n[run] timed out after ${TIMEOUT_MS}ms`; child.kill('SIGKILL'); }, TIMEOUT_MS);
+ child.on('close', (code, signal) => {
+ clearTimeout(timer);
+ resolve({ file, ok: code === 0, code: code ?? signal, ms: Date.now() - started, output });
+ });
+ });
+}
+
+(async () => {
+ if (!files.length) { console.error('no test files matched'); process.exit(1); }
+ const results = [];
+ for (const f of files) {
+ const r = await runOne(f);
+ results.push(r);
+ console.log(`${r.ok ? '✓' : '✗'} ${r.file.padEnd(32)} ${String(r.ms).padStart(6)}ms`);
+ }
+ const failed = results.filter((r) => !r.ok);
+ for (const r of failed) {
+ console.log(`\n── ${r.file} (exit ${r.code}) ──`);
+ console.log(r.output.split('\n').slice(-60).join('\n'));
+ }
+ console.log(`\n${results.length - failed.length}/${results.length} test files passed`);
+ process.exit(failed.length ? 1 : 0);
+})();
new file mode 100644
new file mode 100644
@@ -0,0 +1,85 @@
@@ -0,0 +1,85 @@
+'use strict';
+/** The in-process scheduler, failure backoff, and the outbox relay to OpenVibe.Events. */
+const assert = require('assert');
+const http = require('http');
+const { validate } = require('openvibe-contracts');
+const { boot, site, sourceDef, rss, sleep, suite } = require('./helpers');
+
+const t = suite('scheduler-events');
+const FEED = rss([{ guid: 's1', title: 'Scheduled', link: 'https://example.org/s1' }]);
+
+async function waitFor(fn, ms = 3000) {
+ const end = Date.now() + ms;
+ while (Date.now() < end) { const v = fn(); if (v) return v; await sleep(25); }
+ throw new Error('timed out waiting');
+}
+
+t('due, enabled sources are fetched on schedule; disabled and manual ones never are', async () => {
+ let status = 200;
+ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => (status === 200 ? { body: FEED } : { status }) });
+ const svc = await boot({ worker: 'on' });
+ try {
+ svc.registry.create(sourceDef({ key: 'sched', endpoints: [`${s.origin}/f`], poll_interval_sec: 3600 }), 'test');
+ svc.registry.create(sourceDef({ key: 'off', endpoints: [`${s.origin}/f`], enabled: false }), 'test');
+ svc.registry.create({ key: 'hand', name: 'Manual', type: 'manual', category: 'coupons', terms_note: 'x', enabled: true }, 'test');
+ await waitFor(() => svc.db.prepare("SELECT 1 FROM fetch_runs WHERE source_key = 'sched'").get());
+ const row = svc.registry.get('sched');
+ assert.strictEqual(row.last_state, 'ok');
+ assert.ok(row.next_due_at - Date.now() > 3500 * 1000, 'next poll one interval later');
+ await sleep(200);
+ assert.strictEqual(svc.db.prepare("SELECT COUNT(*) AS n FROM fetch_runs WHERE source_key IN ('off', 'hand')").get().n, 0);
+ assert.strictEqual(svc.db.prepare("SELECT COUNT(*) AS n FROM fetch_runs WHERE source_key = 'sched'").get().n, 1, 'not refetched before it is due');
+
+ // a failing source backs off exponentially
+ status = 500;
+ svc.db.prepare("UPDATE sources SET next_due_at = 0, last_request_at = NULL WHERE key = 'sched'").run();
+ await waitFor(() => svc.registry.get('sched').consecutive_failures === 1);
+ const f1 = svc.registry.get('sched').next_due_at - svc.registry.get('sched').last_run_at;
+ svc.db.prepare("UPDATE sources SET next_due_at = 0, last_request_at = NULL WHERE key = 'sched'").run();
+ await waitFor(() => svc.registry.get('sched').consecutive_failures === 2);
+ const f2 = svc.registry.get('sched').next_due_at - svc.registry.get('sched').last_run_at;
+ assert.strictEqual(f1, 3600 * 1000);
+ assert.strictEqual(f2, 2 * 3600 * 1000);
+
+ // not_before (Retry-After) holds the scheduler back
+ svc.db.prepare("UPDATE sources SET next_due_at = 0, not_before = ?, last_request_at = NULL WHERE key = 'sched'").run(Date.now() + 60000);
+ const n = s.hits('/f').length;
+ await sleep(300);
+ assert.strictEqual(s.hits('/f').length, n);
+ } finally { await svc.stop(); await s.close(); }
+});
+
+t('events are valid envelopes and reach OpenVibe.Events through the relay', async () => {
+ const received = [];
+ const events = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', c => chunks.push(c));
+ req.on('end', () => {
+ const body = JSON.parse(Buffer.concat(chunks).toString('utf8'));
+ received.push(...(body.events || [body]));
+ res.statusCode = 201;
+ res.end('{}');
+ });
+ });
+ await new Promise(r => events.listen(0, '127.0.0.1', r));
+ const s = await site({ '/robots.txt': () => ({ status: 404 }), '/f': () => ({ body: FEED }), '/bad': () => ({ status: 502 }) });
+ const tokenClient = { async authHeaders() { return { Authorization: 'Bearer relay' }; }, invalidate() {} };
+ const svc = await boot({ env: { EVENTS_URL: `http://127.0.0.1:${events.address().port}` }, tokenClient });
+ try {
+ svc.registry.create(sourceDef({ key: 'ev', endpoints: [`${s.origin}/f`, `${s.origin}/bad`] }), 'test');
+ await svc.ingest.run('ev', { trigger: 'manual' });
+ for (let i = 0; i < 5 && svc.outbox.pending(); i++) await svc.relay.flush();
+ assert.strictEqual(svc.outbox.pending(), 0);
+ assert.deepStrictEqual(received.map(e => e.event_type).sort(), ['sources.fetch.failed', 'sources.item.created']);
+ for (const e of received) {
+ const v = validate('events.event-envelope@1', e);
+ assert.ok(v.valid, JSON.stringify(v.errors));
+ assert.strictEqual(e.source, 'sources');
+ assert.strictEqual(e.visibility, 'internal');
+ }
+ const failed = received.find(e => e.event_type === 'sources.fetch.failed');
+ assert.deepStrictEqual([failed.payload.state, failed.payload.http_status, failed.payload.error_code], ['http_error', 502, 'http_502']);
+ } finally { await svc.stop(); await s.close(); await new Promise(r => events.close(() => r())); }
+});
+
+t.run();