generate integrations reference from catalog#2563
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds catalog-backed generation and validation for the integrations reference documentation, along with a CLI script and tests to keep the generated markdown in sync.
Changes:
- Introduce
specify_cli.catalog_docshelpers to render/update the generated integrations table fromintegrations/catalog.json. - Add
scripts/generate_integrations_reference.pywith--check/--writemodes for CI and local updates. - Regenerate
docs/reference/integrations.mdwith generated-table markers and add tests to enforce consistency.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| tests/test_catalog_docs.py | Adds tests asserting the committed docs match the generator and that registry metadata is reflected. |
| src/specify_cli/catalog_docs.py | Implements catalog loading, table rendering, and marker-based replacement for the docs page. |
| scripts/generate_integrations_reference.py | Provides a CLI entrypoint to check or rewrite the generated integrations reference file. |
| docs/reference/integrations.md | Converts the integrations table into a generated block and updates surrounding instructions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
DyanGalih
changed the title
There was a problem hiding this comment.
Love it, but can we integrate it into specify integration search --markdown? Specifically can we keep it simpler than this and just render out the table. On the project end we will take care of integrating it into the docs. I do not want to burden the CLI with that part of the process
|
Thanks for the direction! I've refactored based on your feedback: What changed:
Usage: Prints the full integrations reference table to stdout. The docs team can paste it wherever needed. |
…-markdown - Remove standalone scripts/generate_integrations_reference.py - Strip doc injection machinery from catalog_docs.py; keep only table rendering - Wire render_integrations_table() into existing --markdown flag of integration search - Remove old simple markdown table block from integration_search (was Name|ID|Version|Description|Author) - Simplify tests: drop subprocess/doc-path tests, keep table rendering and metadata tests - Clean up docs/reference/integrations.md: remove generated markers, update note
DyanGalih
force-pushed
the
002-generate-integrations-docs
branch
from
1bdc359 to
59c134c
Compare
…search - Warn when --markdown is combined with filters (query/--tag/--author) which are silently ignored; catch ValueError/FileNotFoundError and surface clean error via console instead of raw traceback (r3244821516) - Add coverage enforcement in list_integrations_for_docs(): raises ValueError with actionable message if any registry key is missing from INTEGRATION_DOC_URLS, preventing silently incomplete doc tables (r3244821589) - Rename test to accurately reflect sources: label derives from registry config, URL comes from INTEGRATION_DOC_URLS doc map — not solely from registry (r3244821607) - Simplify test dict construction to idiomatic dict comprehension (r3244821619)
| def test_integrations_table_renders(): | ||
| table = render_integrations_table() | ||
| assert "| Agent" in table | ||
| assert "| Key" in table | ||
| assert "| Notes" in table | ||
|
|
||
|
|
||
| def test_integrations_reference_label_derives_from_registry_url_from_doc_map(): | ||
| rows = {key: (label, url) for key, label, url, _notes in list_integrations_for_docs()} |
There was a problem hiding this comment.
Fixed. Added test_integrations_reference_doc_is_in_sync to tests/test_catalog_docs.py. It reads INTEGRATIONS_REFERENCE_PATH (an absolute Path constant exported from catalog_docs.py) and asserts that render_integrations_table() output is present as a substring of the committed file, with a clear error message pointing to specify integration search --markdown when out of sync.
| def _get_integration_registry() -> dict[str, Any]: | ||
| from specify_cli.integrations import INTEGRATION_REGISTRY | ||
|
|
||
| return INTEGRATION_REGISTRY | ||
|
|
||
|
|
||
| def list_integrations_for_docs() -> list[tuple[str, str, str | None, str]]: | ||
| registry = _get_integration_registry() |
There was a problem hiding this comment.
Fixed. Updated the module docstring to clarify it derives rows from INTEGRATION_REGISTRY plus hardcoded URL/notes maps (not from integrations/catalog.json). The test file docstring was also updated from "catalog-backed documentation generation" to "integration registry documentation generation". The PR description was updated to say "backed by INTEGRATION_REGISTRY plus per-key URL and notes maps" instead of "catalog-backed" / "from integrations/catalog.json".
| | [Trae](https://www.trae.ai/) | `trae` | Skills-based integration; skills are installed automatically | | ||
| | [Windsurf](https://windsurf.com/) | `windsurf` | | | ||
| | Generic | `generic` | Bring your own agent — use `--integration generic --integration-options="--commands-dir <path>"` for AI coding agents not listed above | | ||
| Run `specify integration search --markdown` to print this table as markdown. |
There was a problem hiding this comment.
Fixed. The PR description validation section previously listed python scripts/generate_integrations_reference.py --check, which does not exist in this branch. The PR description has been updated so the validation section now reads specify integration search --markdown (the actual CLI command) and pytest tests/test_catalog_docs.py -q. The PR description also no longer claims the table is generated from integrations/catalog.json.
mnriem
left a comment
mnriem
left a comment
There was a problem hiding this comment.
Please address Copilot feedback. Please revert the change to integrations.md as we will setup a separate GitHub actions job for that. Thanks for the great work
…ve sync test (GH Actions job will handle)
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
|
|
||
| INTEGRATIONS_REFERENCE_PATH = ( | ||
| Path(__file__).parent.parent.parent / "docs" / "reference" / "integrations.md" | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
Fixed. Removed INTEGRATIONS_REFERENCE_PATH and the from pathlib import Path import from catalog_docs.py — the constant was unused within the module after the sync test was moved to a separate GH Actions job. The module docstring was also updated to remove the reference to it.
| """Search for integrations in the active catalog stack.""" | ||
| if markdown: | ||
| if query or tag or author: | ||
| console.print( | ||
| "[yellow]Warning:[/yellow] --markdown outputs the full built-in integrations table " | ||
| "and ignores query/--tag/--author filters." | ||
| ) |
There was a problem hiding this comment.
Fixed. Updated the integration_search docstring from "Search for integrations in the active catalog stack." to "Search for integrations in the active catalog stack, or output the built-in reference table with --markdown." to reflect the dual behavior.
| from .catalog_docs import render_integrations_table | ||
| try: | ||
| typer.echo(render_integrations_table()) | ||
| except (ValueError, FileNotFoundError) as exc: |
There was a problem hiding this comment.
Fixed. Removed FileNotFoundError from the except clause — render_integrations_table() performs no filesystem I/O and will never raise it. The handler now catches only ValueError (which list_integrations_for_docs() can raise when a registry key is missing from INTEGRATION_DOC_URLS).
| widths = [ | ||
| max(len(header), *(len(_render_cell(row[index])) for row in rows)) | ||
| for index, header in enumerate(("Agent", "Key", "Notes")) | ||
| ] | ||
|
|
||
| def render_row(values: list[str]) -> str: | ||
| return "| " + " | ".join( | ||
| _render_cell(value).ljust(widths[index]) for index, value in enumerate(values) | ||
| ) + " |" | ||
|
|
||
| lines = [ | ||
| render_row(["Agent", "Key", "Notes"]), | ||
| "| " + " | ".join("-" * width for width in widths) + " |", |
There was a problem hiding this comment.
Fixed. Removed the widths/ljust padding entirely from render_integrations_table(). The Agent cell contains full markdown links like [Label](https://...), so computing column widths from the raw cell text made the table extremely wide without any rendering benefit (markdown parsers ignore whitespace padding). Each row is now rendered as "| " + " | ".join(cells) + " |" with a standard --- separator row, which is valid and concise.
…, fix docstring, drop FileNotFoundError
|
Done! Both tasks completed:\n\n- Copilot feedback addressed — all review threads from the latest rounds have been fixed and replied to (removed dead |
| console.print( | ||
| "[yellow]Warning:[/yellow] --markdown outputs the full built-in integrations table " | ||
| "and ignores query/--tag/--author filters." | ||
| ) | ||
| from .catalog_docs import render_integrations_table | ||
| try: | ||
| typer.echo(render_integrations_table()) | ||
| except ValueError as exc: | ||
| console.print(f"[red]Error:[/red] {exc}") |
There was a problem hiding this comment.
Fixed. Both the filter warning and the error message now use typer.echo(..., err=True) so they are written to stderr. The --markdown stdout stream is now a clean Markdown-only output, safe for piping or redirection to a file.
| except ValueError as exc: | ||
| console.print(f"[red]Error:[/red] {exc}") | ||
| raise typer.Exit(1) |
There was a problem hiding this comment.
Fixed. The error path now uses typer.echo(f"Error: {exc}", err=True) — plain text (no Rich markup interpolation), so any [/] characters in exception messages are treated as literals rather than Rich markup tags. Errors also go to stderr, keeping stdout clean for the Markdown output.
| def test_integrations_reference_label_derives_from_registry_url_from_doc_map(): | ||
| rows = {key: (label, url) for key, label, url, _notes in list_integrations_for_docs()} | ||
| assert rows["copilot"][0] == "GitHub Copilot" | ||
| assert rows["copilot"][1] == "https://code.visualstudio.com/" | ||
| assert rows["codex"][0] == "Codex CLI" |
There was a problem hiding this comment.
Fixed. Renamed to test_integrations_docs_label_and_url_sources — shorter, and more clearly signals what's being verified (where labels and URLs come from) without encoding the implementation detail in the name.
| f"Integration(s) missing from INTEGRATION_DOC_URLS: {', '.join(sorted(missing))}. " | ||
| "Add each key to INTEGRATION_DOC_URLS in catalog_docs.py (use None if no URL applies)." | ||
| ) | ||
|
|
There was a problem hiding this comment.
Fixed. list_integrations_for_docs() now also checks for stale entries — any key present in INTEGRATION_DOC_URLS, INTEGRATION_LABEL_OVERRIDES, or INTEGRATION_NOTES that is no longer in the registry raises a ValueError with an actionable message listing the stale keys and pointing to the dicts to clean up. This mirrors the existing missing-key check and keeps both maps in sync with the registry in both directions.
| def _render_cell(value: str) -> str: | ||
| value = value.replace("\r\n", " ").replace("\r", " ").replace("\n", " ") | ||
| return value.replace("|", "\\|") |
There was a problem hiding this comment.
Fixed. Added test_render_cell_escapes_pipes_and_normalizes_newlines which directly exercises _render_cell with values containing |, \n, \r\n, and \r, asserting pipes are escaped as \| and all newline variants are normalized to spaces, preventing regressions in table formatting.
| assert "| Agent" in table | ||
| assert "| Key" in table | ||
| assert "| Notes" in table |
There was a problem hiding this comment.
Fixed. test_integrations_table_renders now asserts the exact first two lines of the output: lines[0] == "| Agent | Key | Notes |" and lines[1] == "| --- | --- | --- |". This verifies the header structure precisely and won't false-positive on matching text inside data rows.
…n header assertion
|
|
||
| from __future__ import annotations | ||
|
|
||
| from specify_cli.catalog_docs import _render_cell, list_integrations_for_docs, render_integrations_table |
There was a problem hiding this comment.
Fixed. Promoted _render_cell to a public render_cell function (removed leading underscore) with a docstring explaining its purpose. The test now imports the public name and is no longer brittle to internal refactors of a private helper.
| def test_render_cell_escapes_pipes_and_normalizes_newlines(): | ||
| assert _render_cell("a|b") == "a\\|b" | ||
| assert _render_cell("a\nb") == "a b" | ||
| assert _render_cell("a\r\nb") == "a b" | ||
| assert _render_cell("a\rb") == "a b" | ||
| assert _render_cell("a|b\nc") == "a\\|b c" |
| def test_integrations_docs_label_and_url_sources(): | ||
| rows = {key: (label, url) for key, label, url, _notes in list_integrations_for_docs()} | ||
| assert rows["copilot"][0] == "GitHub Copilot" | ||
| assert rows["copilot"][1] == "https://code.visualstudio.com/" | ||
| assert rows["codex"][0] == "Codex CLI" | ||
| assert rows["codex"][1] == "https://github.com/openai/codex" |
There was a problem hiding this comment.
Fixed. Refactored test_integrations_docs_label_and_url_sources to mock both _get_integration_registry() and the doc maps (INTEGRATION_DOC_URLS, INTEGRATION_LABEL_OVERRIDES, INTEGRATION_NOTES) using unittest.mock.patch. The test now uses a minimal fake registry with only "copilot" and "codex" entries, so it focuses on testing the renderer and mapping logic without coupling to the live registry's evolving contents. Future registry changes won't break this test unless the mapping logic itself changes.
| INTEGRATION_DOC_URLS, | ||
| INTEGRATION_LABEL_OVERRIDES, |
There was a problem hiding this comment.
Fixed. Removed unused INTEGRATION_DOC_URLS and INTEGRATION_LABEL_OVERRIDES imports from the test file — they were only imported but never used.
|
|
||
| def render_cell(value: str) -> str: | ||
| r"""Escape markdown special characters (pipes) and normalize newlines to spaces. | ||
|
|
There was a problem hiding this comment.
Fixed. Removed trailing whitespace from the blank line in the render_cell() docstring (line 67).
| with patch("specify_cli.catalog_docs._get_integration_registry", return_value=fake_registry): | ||
| with patch("specify_cli.catalog_docs.INTEGRATION_DOC_URLS", fake_doc_urls): | ||
| with patch("specify_cli.catalog_docs.INTEGRATION_LABEL_OVERRIDES", fake_label_overrides): | ||
| with patch("specify_cli.catalog_docs.INTEGRATION_NOTES", fake_notes): | ||
| rows = {key: (label, url) for key, label, url, _notes in list_integrations_for_docs()} | ||
| assert rows["copilot"][0] == "GitHub Copilot" | ||
| assert rows["copilot"][1] == "https://code.visualstudio.com/" | ||
| assert rows["codex"][0] == "Codex CLI" | ||
| assert rows["codex"][1] == "https://github.com/openai/codex" |
There was a problem hiding this comment.
Fixed. Flattened the deeply nested with patch(...) blocks into a single compound with statement using implicit continuation. This reduces indentation and keeps the test body more readable and maintainable.
| registry = _get_integration_registry() | ||
| registry_keys = set(registry) | ||
|
|
||
| missing = [key for key in registry if key not in INTEGRATION_DOC_URLS] |
There was a problem hiding this comment.
Fixed. Changed the missing calculation to derive from registry_keys (which is already computed) instead of iterating registry again: missing = [key for key in registry_keys if key not in INTEGRATION_DOC_URLS]. This eliminates duplicate iteration and clarifies intent.
…pace, optimize missing calculation
| with ( | ||
| patch("specify_cli.catalog_docs._get_integration_registry", return_value=fake_registry), | ||
| patch("specify_cli.catalog_docs.INTEGRATION_DOC_URLS", fake_doc_urls), | ||
| patch("specify_cli.catalog_docs.INTEGRATION_LABEL_OVERRIDES", fake_label_overrides), | ||
| patch("specify_cli.catalog_docs.INTEGRATION_NOTES", fake_notes), | ||
| ): |
There was a problem hiding this comment.
Fixed. Converted the parenthesized context managers to traditional multi-line syntax using explicit patch variables: with patch_registry, patch_urls, patch_labels, patch_notes:. This maintains Python 3.11+ compatibility while improving clarity and addressing the concern.
| missing = [key for key in registry_keys if key not in INTEGRATION_DOC_URLS] | ||
| if missing: | ||
| raise ValueError( | ||
| f"Integration(s) missing from INTEGRATION_DOC_URLS: {', '.join(sorted(missing))}. " | ||
| "Add each key to INTEGRATION_DOC_URLS in catalog_docs.py (use None if no URL applies)." | ||
| ) | ||
|
|
||
| stale: set[str] = ( | ||
| (set(INTEGRATION_DOC_URLS) - registry_keys) | ||
| | (set(INTEGRATION_LABEL_OVERRIDES) - registry_keys) | ||
| | (set(INTEGRATION_NOTES) - registry_keys) | ||
| ) | ||
| if stale: | ||
| raise ValueError( | ||
| f"Stale key(s) in doc maps no longer present in registry: {', '.join(sorted(stale))}. " | ||
| "Remove them from INTEGRATION_DOC_URLS / INTEGRATION_LABEL_OVERRIDES / INTEGRATION_NOTES." | ||
| ) |
There was a problem hiding this comment.
Fixed. Refactored list_integrations_for_docs() to make validation non-fatal at runtime:
- Integrations missing from
INTEGRATION_DOC_URLSare now skipped with a warning (viawarnings.warn) instead of raising an exception - The function gracefully handles missing entries by defaulting URL to None and notes to empty string
- This allows
--markdownto work in environments with varied/extended registries - Strict validation is still enforced in tests via mocked registries with exact doc map coverage
| if markdown: | ||
| if query or tag or author: | ||
| typer.echo( | ||
| "Warning: --markdown outputs the full built-in integrations table " | ||
| "and ignores query/--tag/--author filters.", | ||
| err=True, | ||
| ) | ||
| from .catalog_docs import render_integrations_table | ||
| try: | ||
| typer.echo(render_integrations_table()) | ||
| except ValueError as exc: | ||
| typer.echo(f"Error: {exc}", err=True) | ||
| raise typer.Exit(1) | ||
| return |
There was a problem hiding this comment.
Fixed. Added 3 CLI-level tests for integration search --markdown:
test_cli_integration_search_markdown_success: Verifies the command outputs a valid markdown table to stdout with exit code 0test_cli_integration_search_markdown_with_filters_warns: Verifies warnings about ignored filters are emitted to stderr, not stdouttest_cli_integration_search_markdown_stdout_is_clean: Verifies stdout contains only the markdown table with no error messages
These tests use Typer's CliRunner and exercise both success and warning paths.
| def list_integrations_for_docs() -> list[tuple[str, str, str | None, str]]: | ||
| """List integrations with their documentation URLs and notes. | ||
|
|
||
| Skips any integrations not in INTEGRATION_DOC_URLS (logs warning if any are missing). |
There was a problem hiding this comment.
Fixed. Updated the docstring to say "emits a Python warning" instead of "logs warning" to accurately reflect the use of warnings.warn() rather than logging.
| stdout = result.stdout | ||
| # Stdout should start with the markdown table header | ||
| assert stdout.startswith("| Agent | Key | Notes |") | ||
| # Stdout should not contain any error or warning messages | ||
| assert "Error" not in stdout | ||
| assert "error" not in stdout.lower() |
There was a problem hiding this comment.
Fixed. Improved test_cli_integration_search_markdown_stdout_is_clean() to:
- Assert the markdown header format explicitly (
lines[0] == "| Agent | Key | Notes |") - Verify minimum line count (header + separator + at least one row)
- Check for errors in stderr instead of stdout, avoiding brittleness to legitimate table content
This is more robust and doesn't risk false positives if the word "error" appears in table notes.
| from .catalog_docs import render_integrations_table | ||
| try: | ||
| typer.echo(render_integrations_table()) | ||
| except ValueError as exc: |
There was a problem hiding this comment.
Fixed. Broadened the exception handling from except ValueError to except Exception so that other rendering failures (e.g., KeyError, TypeError, AttributeError) are caught and converted to a clean CLI error message instead of bubbling up as a traceback.
| result = runner.invoke(app, ["integration", "search", "test-query", "--markdown", "--tag", "some-tag"]) | ||
| assert result.exit_code == 0 | ||
| # Warning should be on stderr, table should be on stdout | ||
| assert "Warning" in result.stderr or "ignores" in result.stderr |
There was a problem hiding this comment.
Fixed. Updated the assertion to check for the specific Typer warning message ("ignores query/--tag/--author filters") instead of the generic "Warning" substring, which prevents false positives from unrelated Python warnings.
| # Warn if there are integrations missing from INTEGRATION_DOC_URLS, but don't fail | ||
| missing = sorted(registry_keys - set(INTEGRATION_DOC_URLS)) | ||
| if missing: | ||
| import warnings | ||
| warnings.warn( | ||
| f"Integration(s) missing from INTEGRATION_DOC_URLS: {', '.join(missing)}. " | ||
| "These will be skipped in the docs table. Add them to INTEGRATION_DOC_URLS in catalog_docs.py.", | ||
| stacklevel=2 | ||
| ) |
There was a problem hiding this comment.
Fixed. Added an optional warn_on_missing: bool = False parameter to list_integrations_for_docs(). By default, warnings are disabled, keeping CLI output clean. This allows docs generation code to explicitly opt-in to warnings if needed while the CLI remains quiet.
| typer.echo(f"Error: {exc}", err=True) | ||
| raise typer.Exit(1) |
There was a problem hiding this comment.
Fixed. Improved the exception handler to:
- Use
raise typer.Exit(code=1) from excto preserve the exception chain for debugging - Provide a more contextual error message (
"Error rendering integrations table") instead of just the raw exception string
This ensures the traceback chain is preserved for debuggability while still providing a clean error message to end users.
| def test_cli_integration_search_markdown_success(): | ||
| """Test that `integration search --markdown` outputs the markdown table.""" | ||
| result = runner.invoke(app, ["integration", "search", "--markdown"]) | ||
| assert result.exit_code == 0 | ||
| lines = result.stdout.splitlines() | ||
| assert len(lines) > 2 # At least header, separator, and one data row | ||
| assert lines[0] == "| Agent | Key | Notes |" | ||
| assert lines[1] == "| --- | --- | --- |" |
There was a problem hiding this comment.
Fixed. All three CLI tests now patch _get_integration_registry and the doc maps using a _get_mocked_cli_runner() helper, making them deterministic and independent of the real registry state. Each test now uses explicit start/stop to manage the patches, ensuring consistent behavior regardless of active registrations.
| if key not in INTEGRATION_DOC_URLS: | ||
| continue | ||
|
|
||
| config = integration.config if isinstance(integration.config, dict) else {} |
There was a problem hiding this comment.
Fixed. Replaced integration.config attribute access with getattr(integration, "config", {}) to handle cases where integration objects may not have a .config attribute. The code now gracefully defaults to an empty dict and continues type-checking, making it resilient to different integration implementations.
What changed
INTEGRATION_REGISTRYplus per-key URL and notes maps incatalog_docs.py.docs/reference/integrations.mdso the supported agent table is generated from the integration registry (not hand-maintained).Why
The integrations reference had been hand-maintained, which made it easy for the docs to drift from the runtime registry. This change makes the doc a checked artifact and reduces maintenance overhead.
User impact
specify integration search --markdown.Validation
specify integration search --markdownpytest tests/test_catalog_docs.py -q