Product Feed: Checking Usage Rights Before Connecting

Before connecting a product feed, obtain evidence that precisely authorizes the collection, transformation, retention, and intended use. A public URL, a valid account, a responding endpoint, or a permissive robots.txt is not enough. Link each processing step to a license clause, terms of service, a supplier contract, or written authorization, then note the limits on frequency, territory, duration, and redistribution. If an essential right remains ambiguous, block the automation and ask the supplier for confirmation or seek appropriate legal advice. This verification protects both the source and the project's reversibility.

Separating five commonly confused questions

The first mistake is reducing usage rights to "can we download the file?" Technical access is only one layer. A clean connection distinguishes at least five questions:

1. Access: were the account, key, path, or export provided for this use? 2. Collection: is automated downloading authorized, at what frequency, and within what limits? 3. Transformation: can columns be normalized, indicators calculated, or products matched? 4. Retention: what data can be stored, for how long, and with what deletion measures? 5. Distribution: can titles, images, prices, or stock levels be shown to users, exported, or passed on to a third party?

A positive answer to one question does not carry over to the others. A supplier may authorize import into an internal tool without authorizing the public republication of its images. It may also allow a manual export without providing for automated querying. Only the applicable text and, if necessary, an explicit confirmation, can settle the matter.

Building an evidence file per source

The file must be attached to a clearly identified source. Avoid general notes such as "authorized catalog." Record the supplier's name, the department concerned, the account type, the document, its version or date, the official URL, and the person who validated the interpretation. A screenshot alone is fragile if the text can be downloaded or archived in its original format.

The form of the feed does not change this obligation. The technical differences between CSV, XML, API, and FTP help build the connection, but none of these technologies grants a right of reuse on its own.

What the EU Database Directive changes

Directive 96/9/EC governs the legal protection of databases within the European Union. Among other things, it provides, under certain conditions, for a right covering the extraction or re-utilization of the whole or a substantial part of the content when a substantial investment has been made. It also addresses certain repeated and systematic acts on non-substantial parts. Applying it to a specific catalog, however, depends on the facts, national law, and the contract.

This directive is neither an access license nor an automatic ban on all imports. It shows why "the pages are visible" or "we only take a few lines at a time" is not sufficient as an analysis. Contractual rights, any copyright in the content, database protection, and other applicable rules must be examined separately. In case of real stakes or an ambiguous clause, validation belongs to a competent legal professional; this article provides a documentary method, not legal advice.

Why robots.txt does not settle the question

RFC 9309 formalizes the Robots Exclusion Protocol. It states that the published rules are requested of robots crawling URIs and explicitly specifies that they do not constitute a form of access authorization. The file therefore serves to respect the service's technical preferences, but it replaces neither authentication, nor a license, nor terms of service.

The practical consequence is twofold. A path allowed by robots.txt does not grant the right to copy or republish its content. A disallowed path must be respected by the robot concerned, even if the team believes it has a commercial interest. If a specific contract seems to contradict a technical restriction, do not try to work around it: ask the supplier for the official channel corresponding to the agreement.

Validation procedure before connecting

Step 1: describe the actual use

Write a concrete sentence: "retrieve this file via this channel, at this frequency, keep these fields, and use them for this purpose." Mention the users and any transmissions. A vague purpose makes it impossible to compare the project against the clauses.

Step 2: gather the applicable texts

Collect the contract, the account's terms of service, the feed license, the technical documentation, and any written exchanges that clarify the authorization. Note their date and scope. Marketing pages may explain the service but do not necessarily replace the contractual terms.

Step 3: fill in the rights matrix

For each action — collect, transform, store, display, export, delete — cite the evidence and classify the result: authorized, prohibited, conditional, or undetermined. Do not use "authorized" without a reference. An undetermined box becomes a question to send to the supplier.

Step 4: translate the limits into safeguards

Turn confirmed obligations into settings: maximum frequency, excluded fields, retention period, logging, deletion at the end of the contract, and export limits. Documentation and execution must stay consistent. If the permission expires, collection must be able to stop without deleting the evidence needed for the audit.

Step 5: test on a reversible scope

Once rights are established, use a supplier canary batch to check the channel and the limits on a small, controlled scope. This test does not legitimize unauthorized use; it comes after the documentary validation.

Step 6: plan the review

Set the events that require a fresh reading: change to the terms of service, new domain, account change, addition of images, sharing with a third party, increased frequency, new purpose, or termination. An authorization should not be treated as permanent if its context changes.

Handling gray areas without inventing a permission

When a clause is missing, formulate a closed, traceable question for the supplier. For example: "Does our account authorize a daily retrieval of this file for internal analysis, keeping the identifier, price, and stock level?" Add the question of display and export if those uses are planned. A vague commercial answer must be clarified before automation.

Do not copy extra data "just in case." Minimization reduces risk and simplifies deletion. Checking the columns of a wholesale catalog helps identify the fields that are actually needed and those whose purpose is not demonstrated.

Finally, record refusals. A source blocked for lack of authorization is not a technical failure: it is a compliance decision. The status must indicate the reason, the date, and the next possible action, without automatically relaunching the collection.

What ArbitragePro+ can automate / what the seller must verify

The source-rights-registry function is a proposed specification, not a function declared available. It could attach a usage matrix to each source, along with the official documents, their dates, the restrictions, a review deadline, and a blocking state. It could also prevent a collection when a required authorization is missing or expired, while retaining the history of decisions.

The seller must interpret the applicable contracts, obtain permissions, confirm purposes, and seek legal advice when necessary. They also verify that the technical configuration actually respects the authorized frequency, fields, retention, and distribution. No automated registry turns an ambiguous clause into a permission.

Checklist before activation

• The actual use, channel, frequency, and recipients are described.
• The account holder and the authorization provider are identified.
• Terms of service, licenses, and technical documents are dated and archived.
• Collection, transformation, retention, and distribution each have evidence.
• robots.txt rules are respected without being treated as a license.
• Collected fields have an explicit purpose.
• Limits are translated into technical safeguards.
• A stop and deletion procedure exists.
• Changes of purpose trigger a new review.
• Any undetermined area remains blocked until clarified.

Document an authorized source

Open source management in ArbitragePro+

Official sources

• EUR-Lex, Directive 96/9/EC on the legal protection of databases: https://eur-lex.europa.eu/legal-content/FR/TXT/?uri=CELEX%3A31996L0009 (accessed 2026-08-17).
• IETF, RFC 9309 — Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309.html (accessed 2026-08-17).