Re: [PATCH v2 00/25] Introduce meminspect
From: Mukesh Ojha
Date: Mon Mar 16 2026 - 14:17:51 EST
On Sun, Mar 15, 2026 at 09:24:39PM -0500, Bjorn Andersson wrote:
> On Wed, Mar 11, 2026 at 01:45:44AM +0530, Mukesh Ojha wrote:
> > First of all, I want to thank Eugene for his excellent work on this
> > series. What began as the Qualcomm Minidump driver from me has now
> > evolved into meminspect. He also presented meminspect a few months ago
> > at Linux Plumbers 2025.
> >
> > Video of the recording is available here for anyone interested:
> > https://www.youtube.com/watch?v=aDZv4-kOLSc
> >
> > Introduction:
> >
> > meminspect is a mechanism which allows the kernel to mark specific
> > memory areas for memory dumping or specific inspection, statistics,
> > usage. Once regions are marked, meminspect keeps an internal list with
> > the regions in a dedicated table. Further, these regions can be
> > accessed using specific API by any interested driver. Regions being
> > marked beforehand, when the system is up and running, there is no need
> > nor dependency on a panic handler, or a working kernel that can dump the
> > debug information. meminspect can be primarily used for debugging. The
> > approach is feasible to work when pstore, kdump, or another mechanism do
> > not. Pstore relies on persistent storage, a dedicated RAM area or
> > flash, which has the disadvantage of having the memory reserved all the
> > time, or another specific non volatile memory. Some devices cannot keep
> > the RAM contents on reboot so ramoops does not work. Some devices do not
> > allow kexec to run another kernel to debug the crashed one. For such
> > devices, that have another mechanism to help debugging, like firmware,
> > kmemdump is a viable solution.
> >
> > meminspect can create a core image, similar with /proc/vmcore, with only
> > the registered regions included. This can be loaded into crash tool/gdb
> > and analyzed. This happens if CRASH_DUMP=y. To have this working,
> > specific information from the kernel is registered, and this is done at
> > meminspect init time, no need for the meminspect users to do anything.
> >
> > This version of the meminspect patch series includes two drivers that
> > make use of it: one is the Qualcomm Minidump, and the other one is the
> > Debug Kinfo backend for Android devices, reworked from this source here:
> > https://android.googlesource.com/kernel/common/+/refs/heads/android-mainline/drivers/android/debug_kinfo.c
> > written originally by Jone Chou <jonechou@xxxxxxxxxx>
> >
> > *** History, motivation and available online resources ***
> >
> > The patch series is based on both minidump and kmemdump previous implementations.
> >
> > After the three RFC kmemdump versions, considering the ML discussions, it was decided to
> > move this into kernel/ directory and rework it into naming it meminspect, as Thomas Gleixner
> > suggested.
> >
> > Initial version of kmemdump and discussion is available here:
> > https://lore.kernel.org/lkml/20250422113156.575971-1-eugen.hristev@xxxxxxxxxx/
> >
> > Kmemdump has been presented and discussed at Linaro Connect 2025,
> > including motivation, scope, usability and feasability.
> > Video of the recording is available here for anyone interested:
> > https://www.youtube.com/watch?v=r4gII7MX9zQ&list=PLKZSArYQptsODycGiE0XZdVovzAwYNwtK&index=14
> >
> > Linaro blog on kmemdump can be found here:
> > https://www.linaro.org/blog/introduction-to-kmemdump/
> >
> > Linaro blog on kmemdump step by stem using minidump backend is available here:
> > https://www.linaro.org/blog/kmemdump-step-by-step-on-qualcomm-automotive-platform/
> >
> > The implementation is based on the initial Pstore/directly mapped zones
> > published as an RFC here:
> > https://lore.kernel.org/all/20250217101706.2104498-1-eugen.hristev@xxxxxxxxxx/
> >
> > The back-end implementation for qcom_minidump is based on the minidump
> > patch series and driver written by Mukesh Ojha, thanks:
> > https://lore.kernel.org/lkml/20240131110837.14218-1-quic_mojha@xxxxxxxxxxx/
> >
> > The RFC v2 version with .section creation and macro annotation kmemdump
> > is available here:
> > https://lore.kernel.org/all/20250724135512.518487-1-eugen.hristev@xxxxxxxxxx/
> >
> > The RFC v3 version with making everything static, which was pretty much rejected due to
> > all reasons discussed on the public ML:
> > https://lore.kernel.org/all/20250912150855.2901211-1-eugen.hristev@xxxxxxxxxx/
> >
> > *** How to use meminspect with minidump backend on Qualcomm platform guide ***
> >
> > Prerequisites:
> > Crash tool compiled with target=ARM64 and minor changes required for
> > usual crash mode (minimal mode works without the patch) **A patch can be
> > applied from here https://p.calebs.dev/1687bc ** This patch will be
> > eventually sent in a reworked way to crash tool.
> >
>
> That patch was written 8 months ago, what's the timeline for landing
> this?
>
> It's not feasible to have every users rebuild crash from source and
> maintain this copy in order to use the tool.
Right, Let me see what I can do to make it acceptable..
>
> > Target kernel must be built with : CONFIG_DEBUG_INFO_REDUCED=n ; this
> > will have vmlinux include all the debugging information needed for crash
> > tool.
> >
> > Also, the kernel requires these as well: CONFIG_MEMINSPECT,
> > CONFIG_CRASH_DUMP and the driver CONFIG_QCOM_MINIDUMP
> >
> > Kernel arguments: Kernel firmware must be set to mode 'mini' by kernel
> > module parameter like this : qcom_scm.download_mode=mini
> >
> > After the kernel boots, and minidump module is loaded, everything is
> > ready for a possible crash.
> >
> > Once the crash happens, the firmware will kick in and you will see on
> > the console the message saying Sahara init, etc, that the firmware is
> > waiting in download mode. (this is subject to firmware supporting this
> > mode, I am using sa8775p-ride board)
> >
> > Example of log on the console:
> > "
> > [...]
> > B - 1096414 - usb: init start
> > B - 1100287 - usb: qusb_dci_platform , 0x19
> > B - 1105686 - usb: usb3phy: PRIM success: lane_A , 0x60
> > B - 1107455 - usb: usb2phy: PRIM success , 0x4
> > B - 1112670 - usb: dci, chgr_type_det_err
> > B - 1117154 - usb: ID:0x260, value: 0x4
> > B - 1121942 - usb: ID:0x108, value: 0x1d90
> > B - 1124992 - usb: timer_start , 0x4c4b40
> > B - 1129140 - usb: vbus_det_pm_unavail
> > B - 1133136 - usb: ID:0x252, value: 0x4
> > B - 1148874 - usb: SUPER , 0x900e
> > B - 1275510 - usb: SUPER , 0x900e
> > B - 1388970 - usb: ID:0x20d, value: 0x0
> > B - 1411113 - usb: ENUM success
> > B - 1411113 - Sahara Init
> > B - 1414285 - Sahara Open
> > "
>
> This doesn't add any specific value, it's just "Device entered ramdump
> mode".
Sure, will remove it.
>
> >
> > Once the board is in download mode, you can use the qdl tool (I
> > personally use edl , have not tried qdl yet)
>
> Is this your or Eugen's comment? Why haven't you tested qdl yet?
It was Eugene's comment, but I get it, I can try QDL and check, if it
works.
>
> >, to get all the regions as
> > separate files. The tool from the host computer will list the regions
> > in the order they were downloaded.
> >
> > Once you have all the files simply use `cat` to put them all together,
> > in the order of the indexes. For my kernel config and setup, here is my
> > cat command : (you can use a script or something, I haven't done that so
> > far):
>
> So these need to be sorted in numerical order, by that number at the end
> of the file name?
>
> Do you manually punch these in? How do we make this user friendly?
Yes, manually.. but I think we can do better. We could make
this more user‑friendly by using the section header and string table in
the md_KELF binary both of which existed in the earlier implementation.
Then, we can write an upstream‑friendly script that reads this KELF
metadata file, checks whether a binary with the registered name is
present, and stitches everything together to form a complete ELF that
the crash tool can consume. Let me know if you have any suggestion..
>
> Regards,
> Bjorn
>
> >
> > `cat md_KELF1.BIN md_Kvmcorein2.BIN md_Kconfig3.BIN \
> > md_Ktotalram4.BIN md_Kcpu_poss5.BIN md_Kcpu_pres6.BIN \
> > md_Kcpu_onli7.BIN md_Kcpu_acti8.BIN md_Kmem_sect9.BIN \
> > md_Kjiffies10.BIN md_Klinux_ba11.BIN md_Knr_threa12.BIN \
> > md_Knr_irqs13.BIN md_Ktainted_14.BIN md_Ktaint_fl15.BIN \
> > md_Knode_sta16.BIN md_K__per_cp17.BIN md_Knr_swapf18.BIN \
> > md_Kinit_uts19.BIN md_Kprintk_r20.BIN md_Kprintk_r21.BIN \
> > md_Kprb22.BIN md_Kprb_desc23.BIN md_Kprb_info24.BIN \
> > md_Kprb_data25.BIN md_Khigh_mem26.BIN md_Kinit_mm27.BIN \
> > md_Kunknown29.BIN md_Kunknown30.BIN md_Kunknown31.BIN \
> > md_Kunknown32.BIN md_Kunknown33.BIN md_Kunknown34.BIN \
> > md_Kunknown35.BIN md_Kunknown37.BIN \
> > md_Kunknown38.BIN md_Kunknown39.BIN md_Kunknown40.BIN \
> > md_Kunknown41.BIN md_Kunknown42.BIN md_Kunknown43.BIN \
> > md_Kunknown44.BIN md_Kunknown45.BIN md_Kunknown46.BIN \
> > md_Kunknown47.BIN md_Kunknown48.BIN md_Kunknown49.BIN \
> > md_Kunknown50.BIN md_Kunknown51.BIN md_Kunknown52.BIN \
> > md_Kunknown53.BIN md_Kunknown54.BIN > ./minidump_image`
> >
> > Once you have the resulted file, use `crash` tool to load it, like this:
> > `./crash --no_modules --no_panic --no_kmem_cache --zero_excluded vmlinux minidump_image`
> >
> > There is also a --minimal mode for ./crash that would work without any patch applied
> > to crash tool, but you can't inspect symbols, etc.
> >
> > Once you load crash you will see something like this :
> > KERNEL: minidump/20260310-235110/vmlinux [TAINTED]
> > DUMPFILE: ./minidump/20260310-235110/minidump_image
> > CPUS: 8 [OFFLINE: 7]
> > DATE: Thu Jan 1 05:30:00 +0530 1970
> > UPTIME: 00:00:27
> > TASKS: 0
> > NODENAME: qemuarm64
> > RELEASE: 7.0.0-rc3-next-20260309-00028-g528b3c656121
> > VERSION: #5 SMP PREEMPT Tue Mar 10 18:18:41 UTC 2026
> > MACHINE: aarch64 (unknown Mhz)
> > MEMORY: 0
> > PANIC: "Kernel panic - not syncing: sysrq triggered crash"
> >
> > crash> log
> > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x514f0014]
> > [ 0.000000] Linux version 7.0.0-rc3-next-20260309-00028-g528b3c656121 (@21e3bca4168f) (aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #5 SMP PREEMPT Tue Mar 10 18:18:41 UTC 2026
> >
> > *** Debug Kinfo backend driver ***
> > I need help with the testing of this driver, Anyone who actually wants
> > to test this, feel free to reply to the patch. we have also written a
> > simple DT binding for the driver.
> >
> > Thanks in advance for the review, and apologies if I missed addressing any comment.
> >
> > -Mukesh
> >
> > Changes in v2: https://lore.kernel.org/lkml/20251119154427.1033475-1-eugen.hristev@xxxxxxxxxx/
> > - Fixed doc warnings
> > - Fixed kernel-test robot warnings.
> > - Took Mike suggestion to remove mark inspect flag for dynamic memory.
> > - Added R-b for printk patch.
> > - Modified some commit messages for clarity.
> > - corrected binding change for debug-kinfo as per Rob suggestion.
> >
> > Changelog for meminspect v1:
> > - rename to meminspect
> > - start on top of v2 actually, with the section and all.
> > - remove the backend thing, change the API to access the table
> > - move everything to kernel/
> > - add dependency to CRASH_DUMP instead of a separate knob
> > - move the minidump driver to soc/qcom
> > - integrate the meminspect better into memblock by using a new memblock flag
> > - minor fixes : use dev_err_probe everywhere, rearrange variable declarations,
> > remove some useless code, etc.
> >
> > Changelog for RFC v3:
> > - V2 available here : https://lore.kernel.org/all/20250724135512.518487-1-eugen.hristev@xxxxxxxxxx/
> > - Removed the .section as requested by David Hildenbrand.
> > - Moved all kmemdump registration(when possible) to vmcoreinfo.
> > - Because of this, some of the variables that I was registering had to be non-static
> > so I had to modify this as per David Hildenbrand suggestion.
> > - Fixed minor things in the Kinfo driver: one field was broken, fixed some
> > compiler warnings, fixed the copyright and remove some useless includes.
> > - Moved the whole kmemdump from drivers/debug into mm/ and Kconfigs into mm/Kconfig.debug
> > and it's now available in kernel hacking, as per Randy Dunlap review
> > - Reworked some of the Documentation as per review from Jon Corbet
> >
> > Changelog for RFC v2:
> > - V1 available here: https://lore.kernel.org/lkml/20250422113156.575971-1-eugen.hristev@xxxxxxxxxx/
> > - Reworked the whole minidump implementation based on suggestions from Thomas Gleixner.
> > This means new API, macros, new way to store the regions inside kmemdump
> > (ditched the IDR, moved to static allocation, have a static default backend, etc)
> > - Reworked qcom_minidump driver based on review from Bjorn Andersson
> > - Reworked printk log buffer registration based on review from Petr Mladek
> >
> > I appologize if I missed any review comments.
> > Patches are sent on top on next-20260309 tag
> >
> > ---
> > Eugen Hristev (21):
> > kernel: Introduce meminspect
> > init/version: Annotate static information into meminspect
> > mm/percpu: Annotate static information into meminspect
> > cpu: Annotate static information into meminspect
> > genirq/irqdesc: Annotate static information into meminspect
> > timers: Annotate static information into meminspect
> > kernel/fork: Annotate static information into meminspect
> > mm/page_alloc: Annotate static information into meminspect
> > mm/show_mem: Annotate static information into meminspect
> > mm/swapfile: Annotate static information into meminspect
> > kernel/vmcore_info: Register dynamic information into meminspect
> > kernel/configs: Register dynamic information into meminspect
> > mm/init-mm: Annotate static information into meminspect
> > panic: Annotate static information into meminspect
> > kallsyms: Annotate static information into meminspect
> > mm/mm_init: Annotate static information into meminspect
> > sched/core: Annotate runqueues into meminspect
> > remoteproc: qcom: Move minidump data structures into its own header
> > soc: qcom: Add minidump backend driver
> > soc: qcom: smem: Add minidump platform device
> > meminspect: Add debug kinfo compatible driver
> >
> > Mukesh Ojha (4):
> > mm/numa: Register node data information into meminspect
> > mm/sparse: Register information into meminspect
> > printk: Register information into meminspect
> > dt-bindings: reserved-memory: Add Google Kinfo Pixel reserved memory
> >
> > Documentation/dev-tools/index.rst | 1 +
> > Documentation/dev-tools/meminspect.rst | 144 +++++++
> > .../bindings/reserved-memory/google,kinfo.yaml | 46 ++
> > MAINTAINERS | 14 +
> > drivers/of/platform.c | 1 +
> > drivers/remoteproc/qcom_common.c | 56 +--
> > drivers/soc/qcom/Kconfig | 13 +
> > drivers/soc/qcom/Makefile | 1 +
> > drivers/soc/qcom/minidump.c | 272 ++++++++++++
> > drivers/soc/qcom/smem.c | 10 +
> > include/asm-generic/vmlinux.lds.h | 13 +
> > include/linux/meminspect.h | 263 ++++++++++++
> > include/linux/soc/qcom/minidump.h | 72 ++++
> > init/Kconfig | 1 +
> > init/version-timestamp.c | 3 +
> > init/version.c | 3 +
> > kernel/Makefile | 1 +
> > kernel/configs.c | 6 +
> > kernel/cpu.c | 5 +
> > kernel/fork.c | 3 +
> > kernel/irq/irqdesc.c | 2 +
> > kernel/kallsyms.c | 9 +
> > kernel/meminspect/Kconfig | 30 ++
> > kernel/meminspect/Makefile | 4 +
> > kernel/meminspect/kinfo.c | 284 +++++++++++++
> > kernel/meminspect/meminspect.c | 471 +++++++++++++++++++++
> > kernel/panic.c | 4 +
> > kernel/printk/printk.c | 11 +
> > kernel/sched/core.c | 2 +
> > kernel/time/timer.c | 2 +
> > kernel/vmcore_info.c | 4 +
> > mm/init-mm.c | 11 +
> > mm/mm_init.c | 2 +
> > mm/numa.c | 2 +
> > mm/page_alloc.c | 2 +
> > mm/percpu.c | 2 +
> > mm/show_mem.c | 2 +
> > mm/sparse.c | 6 +
> > mm/swapfile.c | 2 +
> > 39 files changed, 1725 insertions(+), 55 deletions(-)
> > ---
> > base-commit: 343f51842f4ed7143872f3aa116a214a5619a4b9
> > change-id: 20260311-minidump-v2-eed8da647ce5
> >
> > Best regards,
> > --
> > -Mukesh Ojha
> >
--
-Mukesh Ojha