Skip to content

feat(extensions): sysrq-serial-trigger + kernel-debug-tiers (on-device kernel debugging via serial console)#9776

Open
iav wants to merge 2 commits into
mainfrom
feat/sysrq-serial-trigger
Open

feat(extensions): sysrq-serial-trigger + kernel-debug-tiers (on-device kernel debugging via serial console)#9776
iav wants to merge 2 commits into
mainfrom
feat/sysrq-serial-trigger

Conversation

@iav
Copy link
Copy Markdown
Contributor

@iav iav commented May 7, 2026

Description

Two extensions that, together with the kernel patches already merged as PR #9750 (rk3399 8250_dw) and PR #9760 (mvebu-6.18 8250_dw), form a coherent system for on-device kernel debugging through the serial console on headless ARM boards (NAS-style: helios4, helios64, similar).

Each piece is necessary, none is sufficient on its own:

  • kernel-debug-tiers turns on the kernel-side information without which there is nothing to debug — BTF for hung-task tracebacks, printk timestamps and PRINTK_CALLER, lockup/hung-task detection, pstore/ramoops for post-reboot dmesg, KGDB symbol resolution. Three cumulative tiers (KERNEL_DEBUG_TIER=1|2|3, default 1) so the operator picks the cost they're willing to pay; tier 0 is a deliberate no-op.

  • sysrq-serial-trigger turns on the serial console as a debug surface — MAGIC_SYSRQ + MAGIC_SYSRQ_SERIAL_SEQUENCE in kconfig, kernel.sysrq=1 sysctl, and a long enough u-boot BOOTDELAY for the operator to actually catch the prompt. Without this, even a kernel full of debug info can only be examined post-mortem; nothing can be done about a live hang.

  • The 8250_dw driver patches (PR fix(kernel/rockchip64-7.0): restore SysRq-via-BREAK on dw-apb-uart (8250_dw) #9750/fix(kernel/mvebu-6.18): restore SysRq-via-BREAK on dw-apb-uart (8250_dw) #9760, already merged) close the last gap on rk3399 and mvebu: the BREAK condition on a dw-apb-uart goes through dw8250_handle_irq(), which used to swallow LSR.BI without invoking uart_handle_break()handle_sysrq_from_serial(). Before those patches, even a fully configured SysRq-over-serial setup on these SoCs only appeared to work; the operator's BREAK never reached the SysRq dispatcher.

Together these three changes make the well-documented "send BREAK, type the magic sequence, then press a SysRq command" workflow actually functional on rk3399 and mvebu boards, with enough kernel debug info behind it that the resulting tracebacks and KGDB sessions are informative rather than just hex addresses.

Tier guarantees

  • Hard-fail on conflicting BTF preference: KERNEL_BTF=no + KERNEL_DEBUG_TIER>=1 exits early in extension_prepare_config with a helpful message (set KERNEL_BTF=yes/leave unset, or KERNEL_DEBUG_TIER=0).
  • Tier 0 is a real opt-out (extension stays loaded, kernel side off) — useful in shared configs where some boards must keep KERNEL_BTF=no for RAM reasons.
  • sysrq-serial-trigger is independent of the debug tiers; either can be enabled standalone.

Tests performed

End-to-end on helios64 (rockchip64-edge, kernel 7.0.3-rockchip64):

Run Override Expected Result
T1 KERNEL_DEBUG_TIER=1 (default) full kernel build, tier 1 markers + sysrq markers ✓ build OK, kernel 7.0.3-rockchip64, 40m30s
T0 KERNEL_DEBUG_TIER=0 extension loaded, no tier-N kconfig changes
T2 KERNEL_DEBUG_TIER=2 tier 1 + tier 2 (pstore), no tier 3
T3 KERNEL_DEBUG_TIER=3 tier 1 + tier 2 + tier 3 (KGDB)
BTF guard KERNEL_BTF=no (default tier 1) hard-fail in extension_prepare_config ✓ rc=43, exact error message
BTF=no + T0 KERNEL_BTF=no KERNEL_DEBUG_TIER=0 guard skipped, build proceeds

Checklist

Assisted-by: Claude:claude-opus-4.7

Summary by CodeRabbit

  • New Features
    • Multi-tier kernel debugging (tiers 0–3) with validation and BTF handling: tiered diagnostics from no-op to printk/hung-task checks, pstore/ramoops support, and optional KGDB/KDB over serial (requires appropriate boot arguments).
    • Magic SysRq over serial, adjusted autoboot behavior to allow operator interruption and pause, and a sysctl added to enable sysrq on target images.

Review Change Stack

@iav iav requested a review from igorpecovnik as a code owner May 7, 2026 03:26
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 7, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 83179d4b-ecfc-40d3-bfc8-b2317f787f76

📥 Commits

Reviewing files that changed from the base of the PR and between 205685c and 8a8bf53.

📒 Files selected for processing (2)
  • extensions/kernel-debug-tiers.sh
  • extensions/sysrq-serial-trigger.sh
🚧 Files skipped from review as they are similar to previous changes (2)
  • extensions/sysrq-serial-trigger.sh
  • extensions/kernel-debug-tiers.sh

📝 Walkthrough

Walkthrough

Adds two Bash extensions: kernel-debug-tiers (docs, validation/BTF enforcement, and tiered kernel-config hooks for tiers 1–3) and sysrq-serial-trigger (kernel hook enabling Magic SysRq over serial, U-Boot autoboot tweaks, and an image sysctl to enable sysrq).

Changes

Kernel Debug Tiers

Layer / File(s) Summary
File docs and examples
extensions/kernel-debug-tiers.sh
Adds file-level documentation describing KERNEL_DEBUG_TIER semantics, usage examples, and BTF requirements.
Parameter Validation
extensions/kernel-debug-tiers.sh
Validates KERNEL_DEBUG_TIER is 0–3; tier 0 treated as kernel-side no-op with early return; tiers ≥1 enforce KERNEL_BTF=yes or fail and emit alerts.
Tier 1 Kernel Config
extensions/kernel-debug-tiers.sh
Applies printk timestamps/callers, hung-task and soft-lockup detection, stack-end checks, and sets DEFAULT_HUNG_TASK_TIMEOUT=120 when tier ≥1.
Tier 2 Pstore/Ramoops
extensions/kernel-debug-tiers.sh
Enables PSTORE/ramoops-related kernel options (console, RAM, deflate) when tier ≥2.
Tier 3 KGDB/KDB serial
extensions/kernel-debug-tiers.sh
Enables KGDB/KDB serial-console Kconfig options and sets KDB_DEFAULT_ENABLE=0xFF for tier ≥3.

SysRq Serial Trigger

Layer / File(s) Summary
Kernel Configuration
extensions/sysrq-serial-trigger.sh
Enables MAGIC_SYSRQ and MAGIC_SYSRQ_SERIAL; forces MAGIC_SYSRQ_DEFAULT_ENABLE=1; validates non-empty SYSRQ_SERIAL_SEQUENCE and sets MAGIC_SYSRQ_SERIAL_SEQUENCE either in .config or via the hash-only modification list.
U-Boot Configuration
extensions/sysrq-serial-trigger.sh
Sets CONFIG_BOOTDELAY=5 and enables CONFIG_AUTOBOOT_NEVER_TIMEOUT using U-Boot's scripts/config.
Userland Configuration
extensions/sysrq-serial-trigger.sh
Writes /etc/sysctl.d/60-armbian-sysrq.conf with kernel.sysrq = 1 into the image to override distro sysctl defaults.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I tweak configs where kernels play,
Tiers march up and sysrq says hey,
Serial whispers, boot waits five,
A rabbit hops — builds stay alive.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically summarizes the main changes: two new extensions (sysrq-serial-trigger and kernel-debug-tiers) for on-device kernel debugging via serial console, which directly matches the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/sysrq-serial-trigger

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added size/medium PR with more then 50 and less then 250 lines 05 Milestone: Second quarter release Needs review Seeking for review Framework Framework components labels May 7, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@extensions/sysrq-serial-trigger.sh`:
- Around line 48-65: The code accepts an empty SYSRQ_SERIAL_SEQUENCE which, when
set to "", will silently disable serial SysRq; change the logic around seq
(declared from SYSRQ_SERIAL_SEQUENCE) to guard against empty values: if seq is
empty or only whitespace, treat it as unset and skip calling
kernel_config_set_string and avoid appending to kernel_config_modifying_hashes
(or set a safe default sequence), and ensure opts_y/opts_val still reflect
enabling only when a non-empty sequence is provided; update the conditional that
writes MAGIC_SYSRQ_SERIAL_SEQUENCE so kernel_config_set_string and
kernel_config_modifying_hashes are only invoked when seq is non-empty (use the
existing seq variable and the functions kernel_config_set_string and
kernel_config_modifying_hashes).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9620094e-8b87-4446-aa9a-1a0ea730da92

📥 Commits

Reviewing files that changed from the base of the PR and between 7266439 and 30c1112.

📒 Files selected for processing (2)
  • extensions/kernel-debug-tiers.sh
  • extensions/sysrq-serial-trigger.sh

Comment thread extensions/sysrq-serial-trigger.sh
iav added 2 commits May 14, 2026 02:20
Enable Magic SysRq through the serial console for headless ARM boards
(Helios64, etc.). Mainline ships MAGIC_SYSRQ_SERIAL_SEQUENCE empty,
disabling BREAK-triggered SysRq; this extension fills it with a
deliberate sequence after BREAK so the operator on the serial console
can sync/remount-RO/reboot from any kernel state where interrupts flow.
Also overrides distro kernel.sysrq=176 to 1 (full command set) and
extends u-boot autoboot delay so the prompt is actually catchable.

Pair with the companion `kernel-debug-tiers` extension to make the
tracebacks reachable via SysRq actually informative (BTF, hung-task,
KGDB symbol resolution).

Signed-off-by: Igor Velkov <[email protected]>
Cumulative kernel debug-information tiers for headless boards that need
on-device debugging through the serial console:

  KERNEL_DEBUG_TIER=0    no-op (extension loaded but kernel-side disabled)
  KERNEL_DEBUG_TIER=1    printk timestamps + lockup/hung-task detection
                         (default — cheap, no board prerequisites)
  KERNEL_DEBUG_TIER=2    + pstore/ramoops (needs DT or bootarg reservation)
  KERNEL_DEBUG_TIER=3    + KGDB/KDB over serial (needs kgdboc= bootarg)

By itself the extension does nothing visible at runtime; it bakes enough
information into the kernel image that operator-console facilities like
SysRq tracebacks and KGDB sessions are actually useful instead of streams
of hex addresses. Pair with `sysrq-serial-trigger` (separate extension)
to also enable the operator-control surface itself.

extension_prepare_config validates the tier value, hard-fails with a
helpful message on KERNEL_BTF=no + KERNEL_DEBUG_TIER>=1 (BTF is required
for hung-task tracebacks and KGDB symbol resolution; either set
KERNEL_BTF=yes or KERNEL_DEBUG_TIER=0 to opt out of the kernel side),
and forces KERNEL_BTF=yes for tier>=1 when not explicitly set.

Signed-off-by: Igor Velkov <[email protected]>
@iav iav force-pushed the feat/sysrq-serial-trigger branch from 205685c to 8a8bf53 Compare May 13, 2026 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

05 Milestone: Second quarter release Framework Framework components Needs review Seeking for review size/medium PR with more then 50 and less then 250 lines

Development

Successfully merging this pull request may close these issues.

1 participant