generate community extensions index from catalog#2564
Conversation
DyanGalih
changed the title
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR introduces an automated generator for the community extensions reference documentation. The generator reads extensions/catalog.community.json and renders a Markdown table into docs/community/extensions.md between sentinel markers, with a CLI supporting --write and --check modes plus tests verifying drift detection.
Changes:
- Adds a
community_catalog_docshelper module with catalog loading, row iteration, and table/markdown rendering. - Adds a CLI script (
scripts/generate_community_extensions_index.py) supporting print/--write/--checkmodes. - Regenerates
docs/community/extensions.mdfrom the catalog, replacing the manually curated table with generated content between sentinel markers; adds tests for sync, sort, and CLI check mode.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
| src/specify_cli/community_catalog_docs.py | Core helpers to load catalog, iterate rows, and render the generated table/index markdown. |
| scripts/generate_community_extensions_index.py | CLI entry point wrapping the helpers with write/check/print modes. |
| docs/community/extensions.md | Replaces the curated table with a generator-driven section delimited by markers; removes Categories/Effect legends. |
| tests/test_community_catalog_docs.py | Adds tests for sync, alphabetical ordering, and CLI --check behavior. |
Comments suppressed due to low confidence (1)
scripts/generate_community_extensions_index.py:1
--checkand--writeare not declared mutually exclusive, and if both are passed--checksilently wins (write is skipped). Useparser.add_mutually_exclusive_group()or validate explicitly to make the contract clear.
#!/usr/bin/env python3
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def load_community_catalog(path: Path = COMMUNITY_CATALOG_PATH) -> dict[str, Any]: | ||
| """Load and validate the community catalog JSON file.""" | ||
| data = json.loads(path.read_text(encoding="utf-8")) | ||
| if not isinstance(data, dict): | ||
| raise ValueError(f"Expected {path} to contain a JSON object") | ||
| extensions = data.get("extensions") | ||
| if not isinstance(extensions, dict): | ||
| raise ValueError(f"Expected {path} to contain an 'extensions' object") | ||
| return data |
| @@ -1,124 +1,114 @@ | |||
| # Community Extensions | |||
|
|
|||
| > [!NOTE] | |||
| > Community extensions are independently created and maintained by their respective authors. Maintainers only verify that catalog entries are complete and correctly formatted — they do **not review, audit, endorse, or support the extension code itself**. The Community Extensions website is also a third-party resource. Review extension source code before installation and use at your own discretion. | |||
| > Community extensions are independently created and maintained by their respective authors. Maintainers only verify that catalog entries are complete and correctly formatted - they do **not review, audit, endorse, or support the extension code itself**. The Community Extensions website is also a third-party resource. Review extension source code before installation and use at your own discretion. | |||
|
|
||
| 🔍 **Browse and search community extensions on the [Community Extensions website](https://speckit-community.github.io/extensions/).** | ||
|
|
||
| The following community-contributed extensions are available in [`catalog.community.json`](https://github.com/github/spec-kit/blob/main/extensions/catalog.community.json): | ||
|
|
||
| **Categories:** | ||
| <!-- BEGIN GENERATED COMMUNITY EXTENSIONS TABLE --> |
| widths = [ | ||
| max(len(header), *(len(_render_cell(row[index])) for row in table_rows)) | ||
| for index, header in enumerate(("Extension", "ID", "Description", "Tags", "Verified")) | ||
| ] |
There was a problem hiding this comment.
Fixed. _render_cell now normalizes \r\n and \r to spaces (in addition to \n) and escapes literal | as \| to prevent broken markdown table cells. This was already included in the latest push (b47ad1a).
| if start == -1 or end == -1 or end < start: | ||
| raise ValueError(f"Could not find generated table markers in {doc_path}") | ||
|
|
||
| start_end = start + len(GENERATED_START_MARKER) |
| if args.check: | ||
| current = args.doc.read_text(encoding="utf-8") | ||
| if current != generated: | ||
| return 1 | ||
| return 0 |
| def test_community_extensions_generator_check_mode() -> None: | ||
| """The CLI check mode should fail when the committed index drifts.""" | ||
| result = subprocess.run( | ||
| [ | ||
| sys.executable, | ||
| "scripts/generate_community_extensions_index.py", |
| def _format_tags(tags: Any) -> str: | ||
| if not isinstance(tags, list) or not tags: | ||
| return "—" |
There was a problem hiding this comment.
Fixed. list_community_extensions() now returns the raw tags list (data-pure — "tags": ext.get("tags") or []). The _format_tags() call has been moved to render_community_extensions_table() where it belongs in the render layer. This keeps the iteration/sort code free of display concerns and simplifies testing.
| ROOT_DIR = Path(__file__).resolve().parents[2] | ||
| COMMUNITY_CATALOG_PATH = ROOT_DIR / "extensions" / "catalog.community.json" | ||
| COMMUNITY_INDEX_PATH = ROOT_DIR / "docs" / "community" / "extensions.md" |
| if markdown: | ||
| from .community_catalog_docs import render_community_extensions_table | ||
| typer.echo(render_community_extensions_table()) | ||
| return | ||
|
|
||
| from .extensions import ExtensionCatalog, ExtensionError | ||
|
|
||
| project_root = _require_specify_project() |
There was a problem hiding this comment.
Fixed (ea06181). Added a warning when --markdown is combined with any filter options (query, --tag, --author, --verified): the user now sees Warning: --markdown outputs the full community catalog and ignores filters. The --markdown path short-circuits before _require_specify_project() intentionally (it renders the canonical community catalog for doc generation, not the project's installed catalog), so filters are irrelevant — but we now make that explicit rather than silently discarding them.
| ROOT_DIR = Path(__file__).resolve().parents[2] | ||
| COMMUNITY_CATALOG_PATH = ROOT_DIR / "extensions" / "catalog.community.json" |
There was a problem hiding this comment.
Fixed (ea06181). list_community_extensions() now checks COMMUNITY_CATALOG_PATH.exists() before reading and raises a FileNotFoundError with an actionable message: "Community catalog not found: <path>. The --markdown flag requires a spec-kit source checkout." This replaces the raw, confusing FileNotFoundError with a traceback into site-packages that a pip-installed user would otherwise see.
| widths = [ | ||
| max(len(header), *(len(_render_cell(row[index])) for row in table_rows)) | ||
| for index, header in enumerate(headers) | ||
| ] | ||
|
|
||
| def render_row(values: list[str]) -> str: | ||
| return "| " + " | ".join( | ||
| _render_cell(value).ljust(widths[index]) for index, value in enumerate(values) | ||
| ) + " |" | ||
|
|
||
| lines = [ | ||
| render_row(list(headers)), | ||
| "| " + " | ".join("-" * width for width in widths) + " |", |
There was a problem hiding this comment.
Fixed (ea06181). render_community_extensions_table() now uses a simple | value | fixed separator (| --- |) instead of computing per-column widths with ljust padding. Markdown renderers ignore column alignment, and the padded output was inflating file size and diff noise significantly (the old table had cells padded to hundreds of characters).
| def _format_tags(tags: Any) -> str: | ||
| if not isinstance(tags, list) or not tags: | ||
| return "—" | ||
| cleaned = [f"`{str(tag)}`" for tag in tags if str(tag).strip()] | ||
| return ", ".join(cleaned) if cleaned else "—" |
There was a problem hiding this comment.
Fixed (ea06181). _format_tags() now strips | from tag values before wrapping them in backticks: str(tag).replace('|', '').strip(). This prevents the _render_cell pipe-escape from producing \| inside a backtick code span, which would render literally as \| in Markdown rather than being invisible. Tags with pipes will have the pipe silently removed.
| for ext_id, ext in extensions.items(): | ||
| if not isinstance(ext, dict): | ||
| raise ValueError(f"Community extension {ext_id!r} must be a mapping") |
There was a problem hiding this comment.
Fixed (ea06181). The --markdown handler in extension_search now wraps render_community_extensions_table() in a try/except (ValueError, FileNotFoundError) block and surfaces the error via console.print(f"[red]Error:[/red] {exc}") + typer.Exit(1), consistent with the rest of the CLI's error handling pattern.
| The following community-contributed extensions are available in [`catalog.community.json`](https://github.com/github/spec-kit/blob/main/extensions/catalog.community.json): | ||
|
|
||
| **Categories:** | ||
| > Run `specify extension search --markdown` to regenerate this table. |
There was a problem hiding this comment.
Acknowledged. CI sync enforcement (e.g. a pre-commit hook or CI job that runs specify extension search --markdown and diffs against the committed file) is a valid follow-up. It's deferred to a separate PR/issue as it involves workflow/CI infrastructure changes beyond the scope of this feature.
| The following community-contributed extensions are available in [`catalog.community.json`](https://github.com/github/spec-kit/blob/main/extensions/catalog.community.json): | ||
|
|
||
| **Categories:** | ||
| > Run `specify extension search --markdown` to regenerate this table. |
There was a problem hiding this comment.
Intentional. The old hand-maintained table had separate Category and Effect columns, but the community catalog JSON doesn't have structured category/effect fields — it only has free-form tags. Preserving the old legends would mean either duplicating static prose (which drifts) or fabricating a mapping from tags to categories (which adds complexity without a reliable source). The tags column surfaces the same taxonomy signals (e.g. docs, code, process, integration, etc.) without the maintenance burden of curated categories. This is a deliberate trade-off.
| def test_community_extensions_table_renders() -> None: | ||
| table = render_community_extensions_table() |
There was a problem hiding this comment.
Fixed (ea06181). Added 9 edge-case tests using synthetic catalogs (patching COMMUNITY_CATALOG_PATH via unittest.mock.patch):
test_missing_catalog_file—FileNotFoundErrorwith actionable messagetest_malformed_json— raises on invalid JSONtest_non_dict_root—ValueErrorwhen root is not a dicttest_missing_extensions_key—ValueErrorwhenextensionskey is absenttest_non_dict_extension_value—ValueErrorwhen an extension entry is not a mappingtest_empty_catalog_raises—ValueErrorwhen catalog has no extensionstest_extension_without_repository— name rendered as plain text (no link)test_tags_containing_pipe_do_not_break_table— pipe stripped from tag valuestest_non_list_tags_renders_em_dash— non-list tags fall back to—
All 11 tests pass.
| def list_community_extensions() -> list[dict[str, Any]]: | ||
| """Return community extensions sorted alphabetically by name then ID.""" | ||
| data = json.loads(COMMUNITY_CATALOG_PATH.read_text(encoding="utf-8")) |
There was a problem hiding this comment.
Fixed. list_community_extensions() and render_community_extensions_table() now both accept an explicit path: Path = COMMUNITY_CATALOG_PATH parameter. Tests have been updated to pass the path directly (e.g. list_community_extensions(path=tmp_path / \"missing.json\")) instead of monkeypatching the module global, which is the cleaner and more reusable approach the reviewer suggested.
…arkdown - Remove standalone scripts/generate_community_extensions_index.py - Strip doc injection machinery from community_catalog_docs.py; keep only table rendering - Rename iter_community_extensions -> list_community_extensions (public, reflects return type) - Fix _render_cell to also normalize \r\n/\r and escape | as \| (parity with catalog_docs.py) - Wire render_community_extensions_table() into new --markdown flag of extension search - Simplify tests: drop subprocess/doc-path tests, keep table rendering and sort tests - Clean up docs/community/extensions.md: remove generated markers, add regen note
- Restore em dash in docs/community/extensions.md (was accidentally changed to hyphen) - Move _format_tags() call from list_community_extensions() to render layer: list_community_extensions() now returns raw tags list (data-pure); render_community_extensions_table() formats tags at render time
…alog_docs - Remove width-padding from render_community_extensions_table(); use simple '| value |' separators — Markdown ignores column alignment padding and it inflates diff noise considerably (r3244775940) - Strip '|' from tag values in _format_tags() so tags wrapped in backtick code spans don't produce malformed escaped sequences like '\|' inside code spans (r3244775963) - Add FileNotFoundError with actionable message when catalog file is missing (e.g. when package is installed via pip/pipx outside a source checkout) (r3244775915) - Warn when --markdown is combined with filters (query/--tag/--author/--verified) which are silently ignored; catch ValueError/FileNotFoundError and surface clean error via console instead of raw traceback (r3244775855, r3244775979) - Add 9 edge-case tests covering: missing file, malformed JSON, non-dict root, missing extensions key, non-dict extension value, empty catalog, missing repository, tags with pipes, and non-list tags (r3244776046) - Regenerate docs/community/extensions.md with compact table format
DyanGalih
force-pushed
the
003-generate-community-docs
branch
from
ea06181 to
0cb36ea
Compare
| if markdown: | ||
| if query or tag or author or verified: | ||
| console.print( | ||
| "[yellow]Warning:[/yellow] --markdown outputs the full community catalog " | ||
| "and ignores filters (query, --tag, --author, --verified)." | ||
| ) | ||
| from .community_catalog_docs import render_community_extensions_table | ||
| try: | ||
| typer.echo(render_community_extensions_table()) | ||
| except (ValueError, FileNotFoundError) as exc: | ||
| console.print(f"[red]Error:[/red] {exc}") | ||
| raise typer.Exit(1) | ||
| return |
There was a problem hiding this comment.
Acknowledged. The warning is already emitted to stdout before the unfiltered table output. Piping to a file will mix the warning into the table, which is not ideal, but since the warning is printed via console.print to stderr-equivalent and typer.echo outputs the table to stdout they should be separable in practice. Adding --markdown filter rejection with non-zero exit would break the common doc-generation workflow of specify extension search --markdown > file.md. The current behavior (warn + emit) is the chosen trade-off, consistent with how this was also resolved in PR #2563 for integration_search.
| ROOT_DIR = Path(__file__).resolve().parents[2] | ||
| COMMUNITY_CATALOG_PATH = ROOT_DIR / "extensions" / "catalog.community.json" |
There was a problem hiding this comment.
Acknowledged. This is a known limitation documented in the FileNotFoundError message: \"The --markdown flag requires a spec-kit source checkout.\" The flag is intentionally a contributor/maintainer tool for regenerating the reference table, not an end-user feature. Packaging via importlib.resources or shipping the catalog as package data would be a separate improvement tracked as a follow-up.
| def _render_cell(value: str) -> str: | ||
| return value.replace("\r\n", " ").replace("\r", " ").replace("\n", " ").replace("|", "\\|") |
There was a problem hiding this comment.
Fixed. Raw field values (name, description) are now escaped with _render_cell() before composing Markdown link syntax. The render_row() helper no longer re-escapes already-escaped values — it concatenates them directly. This ensures a pipe inside a name or description cannot produce [name\\|...]({url}) which would break the link target.
| if not isinstance(tags, list) or not tags: | ||
| return "—" | ||
| # Strip | from tag values so they don't break table syntax inside backtick spans | ||
| cleaned = [f"`{str(tag).replace('|', '').strip()}`" for tag in tags if str(tag).strip()] |
There was a problem hiding this comment.
Fixed. _format_tags() now cleans first and filters on the cleaned value: [f\"{c}\" for tag in tags if (c := str(tag).replace(\"|\", \"\").strip())]. A tag like \" | \" that previously passed the str(tag).strip() check but would produce `` after pipe removal is now filtered out correctly.
| # pipe stripped from tag — table should render cleanly | ||
| assert "`foobar`" in table | ||
|
|
||
|
|
There was a problem hiding this comment.
Fixed. test_tags_containing_pipe_do_not_break_table now also asserts: (1) the data row is well-formed — foo_row.count(\"|\") == 6 for a 5-column table, and (2) the id field falls back to ext_id when the \"id\" key is absent from the extension entry — assert \"\foo`" in table`.
|
|
||
| **Categories:** | ||
| > Run `specify extension search --markdown` to regenerate this table. | ||
|
|
There was a problem hiding this comment.
Intentional — addressed previously in this thread. The community catalog JSON has only free-form tags, no structured category/effect fields. Preserving the old static legends would require either drift-prone prose duplication or a fabricated tag-to-category mapping. The tags column surfaces the same taxonomy (e.g. docs, code, process, integration) without maintenance overhead. This is a deliberate trade-off.
| The following community-contributed extensions are available in [`catalog.community.json`](https://github.com/github/spec-kit/blob/main/extensions/catalog.community.json): | ||
|
|
||
| **Categories:** | ||
| > Run `specify extension search --markdown` to regenerate this table. |
There was a problem hiding this comment.
Acknowledged — deferred to a follow-up PR/issue. Adding CI enforcement (e.g. a job that runs specify extension search --markdown, diffs against the committed file, and fails on drift) is valid and desirable but involves CI/workflow infrastructure changes beyond the scope of this feature.
| if not rows: | ||
| raise ValueError("Community catalog has no extensions") |
There was a problem hiding this comment.
Acknowledged. Raising on an empty catalog is intentional — an empty community catalog most likely means something went wrong (wrong path, corrupt file, etc.) rather than a legitimate state. Emitting a header-only table would silently replace the doc with content-free output, which is harder to notice. The ValueError keeps it loud and explicit.
| "name": str(ext.get("name") or ext_id), | ||
| "id": str(ext.get("id") or ext_id), |
There was a problem hiding this comment.
Acknowledged — lower priority. The or fallback (ext.get(\"name\") or ext_id) intentionally treats empty string as missing (same behaviour as None), which is the right default since an extension with an empty name should display its key. Falsy non-string values like 0 or False for a name field are not valid per the catalog schema, so the edge case is hypothetical. Adding explicit is None checks or schema validation can be done in a follow-up if schema enforcement is added.
| tag: Optional[str] = typer.Option(None, "--tag", help="Filter by tag"), | ||
| author: Optional[str] = typer.Option(None, "--author", help="Filter by author"), | ||
| verified: bool = typer.Option(False, "--verified", help="Show only verified extensions"), | ||
| markdown: bool = typer.Option(False, "--markdown", help="Output results as a markdown table"), |
There was a problem hiding this comment.
Fixed. Updated to: \"Output the full community catalog as a markdown table (ignores query/tag/author/verified filters)\". The behavior is now explicit from --help without needing to discover the runtime warning.
…log_docs - Parameterize list_community_extensions(path=) and render_community_extensions_table(path=) so callers can supply a custom catalog path without monkeypatching module globals; update tests to use explicit path= args instead of unittest.mock.patch (r3244776065) - Escape raw field values (name, description) with _render_cell() BEFORE composing Markdown link syntax [name](url) so a pipe inside a name/description cannot break a link target; render_row() no longer re-escapes already-escaped values (r3244829318) - Fix _format_tags() filter order: clean (replace+strip) first, then filter on the cleaned value — a tag like ' | ' previously passed the filter but produced an empty backtick span after pipe removal (r3244829330) - Update test_tags_containing_pipe: assert row is well-formed (6 pipe separators for 5-column table), assert id falls back to ext_id when 'id' field is absent (r3244829343) - Change pytest.raises(Exception) to pytest.raises(json.JSONDecodeError) for the malformed JSON test — the broad Exception catch masks unrelated failures (r3244829358) - render_community_extensions_table() now returns a trailing newline so Markdown renderers see a blank line between the last table row and subsequent paragraphs; restore footer paragraph in extensions.md that was previously crammed onto same line as last table row (r3244829373) - Update --markdown help text to make 'ignores filters' behavior explicit rather than relying on the runtime warning alone (r3244829452)
What changed
docs/community/extensions.mdbacked byextensions/catalog.community.json.--checkmode catches drift.Why
Impact
Validation
python scripts/generate_community_extensions_index.py --checkpytest tests/test_community_catalog_docs.py -q