Preview
← BACK

MidnightCrash

Scenario

A production server crashed unexpectedly and rebooted. The crash happened at a strange time, and we doubt it was a simple hardware fault. A kernel crash dump was captured. Your mission is to analyze it to find the real cause of the crash and determine if any other suspicious activity was present on the system.

We are provided with MidnightCrash.zip, it contains:

  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2026-03-16 20:36   MidnightCrash/
109071779  2025-11-04 07:13   MidnightCrash/ubuntu22.04-5.15.0-25-generic-202511032103.kdump
723241568  2025-11-04 07:14   MidnightCrash/vmlinux-dbgsym

Tasks

Task 1

We can start off by installing crash to analyze these files.

crash vmlinux-dbgsym ubuntu22.04-5.15.0-25-generic-202511032103.kdump

# crash 8.0.2
# ...
# GNU gdb (GDB) 10.2
# Copyright (C) 2021 Free Software Foundation, Inc.
# ...
#       KERNEL: vmlinux-dbgsym  [TAINTED]
#     DUMPFILE: ubuntu22.04-5.15.0-25-generic-202511032103.kdump  [PARTIAL DUMP]
#         CPUS: 2
#         DATE: Tue Nov  4 03:03:29 CET 2025
#       UPTIME: 00:38:58
# LOAD AVERAGE: 0.15, 0.08, 0.12
#        TASKS: 548
#     NODENAME: ubuntu-2204
#      RELEASE: 5.15.0-25-generic
#      VERSION: #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022
#      MACHINE: x86_64  (2687 Mhz)
#       MEMORY: 2 GB
#        PANIC: "Oops: 0002 [#1] SMP PTI" (check log for details)
#          PID: 9236
#      COMMAND: "cat"
#         TASK: ffff8e0479346200  [THREAD_INFO: ffff8e0479346200]
#          CPU: 0
#        STATE: TASK_RUNNING (PANIC)

What is the hostname of the crashed server?

Looking at NODENAME we see ubuntu-2204

Task 2

What is the assigned IP address of the server at time of crash?

Using net we see the interface ens33 showing the local IP address: 192.168.1.135

Task 3

When did the server crash? (UTC)

The initial output of sys we see in Task 1, shows the DATE as a CET (Central European Time, which is UTC+1) we can convert and we get: 2025-11-04 02:03:29

Task 4 & 5

Looking at the log we see crash information:

...
[ 2338.667536] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 2338.667546] #PF: supervisor write access in kernel mode
[ 2338.667549] #PF: error_code(0x0002) - not-present page
[ 2338.667551] PGD 0 P4D 0
[ 2338.667556] Oops: 0002 [#1] SMP PTI
[ 2338.667561] CPU: 0 PID: 9236 Comm: cat Kdump: loaded Tainted: G           OE     5.15.0-25-generic #25-Ubuntu
[ 2338.667615] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2338.667619] RIP: 0010:core_helper_read+0x8/0x15 [core_helper]
[ 2338.667633] Code: Unable to access opcode bytes at RIP 0xffffffffc0a93fde.
[ 2338.667635] RSP: 0018:ffff9a64c611fe60 EFLAGS: 00010246
[ 2338.667639] RAX: 0000000000000000 RBX: ffff8e040e909780 RCX: ffff9a64c611fed0
[ 2338.667641] RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: ffff8e04107cb500
[ 2338.667643] RBP: ffff9a64c611fe80 R08: 0000000000000001 R09: ffff8e0419f77ab0
[ 2338.667646] R10: 0000000000020000 R11: 0000000000000000 R12: 0000000000000000
[ 2338.667648] R13: ffff8e04107cb500 R14: ffff9a64c611fed0 R15: 00007fae8be3d000
[ 2338.667650] FS:  00007fae8cc45740(0000) GS:ffff8e047be00000(0000) knlGS:0000000000000000
[ 2338.667653] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2338.667655] CR2: ffffffffc0a93fde CR3: 000000004cdd0004 CR4: 00000000003706f0
[ 2338.667660] Call Trace:
[ 2338.667663]  <TASK>
[ 2338.667667]  ? proc_reg_read+0x5a/0x90
[ 2338.667676]  vfs_read+0x9f/0x1a0
[ 2338.667683]  ksys_read+0x67/0xe0
[ 2338.667685]  __x64_sys_read+0x19/0x20
[ 2338.667688]  do_syscall_64+0x5c/0xc0
[ 2338.667694]  ? exc_page_fault+0x89/0x160
[ 2338.667697]  ? asm_exc_page_fault+0x8/0x30
[ 2338.667704]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 2338.667708] RIP: 0033:0x7fae8cd5c852
[ 2338.667711] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 1a b4 0c 00 e8 a5 1d 02 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24
[ 2338.667715] RSP: 002b:00007fff5f513348 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2338.667718] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007fae8cd5c852
[ 2338.667720] RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: 0000000000000003
[ 2338.667722] RBP: 00007fae8be3d000 R08: 00007fae8be3c010 R09: 00007fae8be3c010
[ 2338.667724] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000022000
[ 2338.667726] R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
[ 2338.667730]  </TASK>
[ 2338.667732] Modules linked in: core_helper(OE) tls rfcomm cpuid binfmt_misc bnep intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock snd_ens1371 snd_ac97_codec gameport ac97_bus snd_pcm nls_iso8859_1 vmw_balloon snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device joydev snd_timer input_leds serio_raw snd btusb btrtl btbcm btintel bluetooth ecdh_generic ecc soundcore vmw_vmci mac_hid sch_fq_codel vmwgfx ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp drm parport ip_tables x_tables autofs4 hid_generic crc32_pclmul usbhid psmouse hid e1000 mptspi mptscsih ahci mptbase libahci scsi_transport_spi pata_acpi i2c_piix4
[ 2338.667900] CR2: 0000000000000000

And we can cross reference the PID 9236 with the ps command, though it's also bottom-most line:

>    9236    4555   0  ffff8e0479346200  RU   0.1    17168     1748  cat

The crash was triggered by a specific process. What was the PID of the active process that caused the panic?

9236

Which command-line utility was leveraged by the previous process to trigger the crash?

cat

Task 6

What was the kernel's fatal panic bug message?

From the previous log we can see the BUG message: kernel NULL pointer dereference, address: 0000000000000000

Task 7

We can look at the virtual memory to try and identify extra information about the process and the crash, we can use vm ffff8e0479346200 where that vmem address comes form the ps earlier:

PID: 9236     TASK: ffff8e0479346200  CPU: 0    COMMAND: "cat"
       MM               PGD          RSS    TOTAL_VM
ffff8e040c7ce640  ffff8e044cdd0000  1748k    17168k
      VMA           START       END     FLAGS FILE
ffff8e0442dc65b0 55df2e870000 55df2e872000 8000071 /usr/bin/cat
ffff8e0442dc6750 55df2e872000 55df2e876000 8000075 /usr/bin/cat
ffff8e0442dc6dd0 55df2e876000 55df2e878000 8000071 /usr/bin/cat
ffff8e0442dc61a0 55df2e878000 55df2e879000 8100071 /usr/bin/cat
ffff8e040d004270 55df2e879000 55df2e87a000 8100073 /usr/bin/cat
...

That's not the process that was called with cat. Let's look further, we can try with files 9236:

PID: 9236     TASK: ffff8e0479346200  CPU: 0    COMMAND: "cat"
ROOT: /    CWD: /root/module
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR  /dev/pts/1
  1 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR  /dev/pts/1
  2 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR  /dev/pts/1
  3 ffff8e04107cb500 ffff8e04072d8000 ffff8e0419f77ab0 REG  /proc/jiffies_ext

What is the absolute path of the malicious file that caused the kernel crash?

/proc/jiffies_ext

Task 8

This is interesting, since cat crashed, let's try to differentiate between a normal activity, from a rootkit attempt.

I suspect rootkit since the user called cat on a /proc file, which is not a normal activity, let's look at the stack trace to understand what happened in the kernel exactly.

What is the name of the function at the top of the kernel's call stack at time of the crash?

From the trace we copied above we see a syscall to read, which ends up calling core_helper_read

Task 9

Looking at the at the end of the log dump, we see a specific core_helper(OE), OE means out-of-tree, this is not normal, let's look at the module list:

crash> mod | head -n 1
     MODULE       NAME                                     BASE          SIZE  OBJECT FILE
crash> mod | grep core_helper
ffffffffc0a96000  core_helper                        ffffffffc0a94000   16384  (not loaded)  [CONFIG_KALLSYMS]

This function belongs to a malicious kernel module. What is the base memory address of this module?

0xffffffffc0a94000

Task 10

Since it's (not loaded) the kernel cannot point us towards a specific file, this is probably an evasion technique, though since the kernel was compiled with CONFIG_KALLSYMS we should have the symbol information on the dump:

crash> sym -m core_helper
ffffffffc0a94000 MODULE START: core_helper
ffffffffc0a94000 (t) core_helper_read
ffffffffc0a94015 (t) core_helper_exit
ffffffffc0a94015 (T) cleanup_module
ffffffffc0a95024 (?) _note_9
ffffffffc0a9503c (?) _note_8
ffffffffc0a950e0 (?) proc_file_ops
ffffffffc0a96000 (?) __this_module
ffffffffc0a98000 MODULE END: core_helper

What is the function name in the malicious kernel module that performs cleanup?

cleanup_module

Task 11

Since we're clearly dealing with a rootkit that crashed the system, either voluntarily or by accident, let's see if there's any sudo processes running that are suspicious:

crash> ps | grep -A 4 sudo
     4552    4504   0  ffff8e0410681880  IN   0.1    23008     3108  sudo
     4553    4552   1  ffff8e040e884980  IN   0.1    23008     1176  sudo
     4554    4553   0  ffff8e040e880000  IN   0.1    21700     2864  su
     4555    4554   0  ffff8e0410680000  IN   0.1    19784     3032  bash
     4639    3968   1  ffff8e040e889880  IN   0.9   502588    18228  update-notifier
     4644    3968   0  ffff8e0410103100  IN   0.9   502588    18228  gmain
--
     8923    4555   1  ffff8e04300e1880  IN   0.3    23004     5840  sudo
     8924    8923   0  ffff8e04300e6200  IN   0.1    23004     2380  sudo
     8925    8924   0  ffff8e041046b100  IN   0.1    17028     1928  httpd-worker
     8931    3778   0  ffff8e04300e4980  IN   7.3  4408992   152508  threaded-ml
     9056       2   0  ffff8e0410674980  ID   0.0        0        0  [kworker/u256:0]
     9230       2   0  ffff8e040dc4c980  ID   0.0        0        0  [kworker/0:1]

crash> files 8925
PID: 8925     TASK: ffff8e041046b100  CPU: 0    COMMAND: "httpd-worker"
ROOT: /    CWD: /root
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff8e040d2c0500 ffff8e040d826780 ffff8e0430529680 CHR  /dev/pts/2
  1 ffff8e040d2c0500 ffff8e040d826780 ffff8e0430529680 CHR  /dev/pts/2
  2 ffff8e040d2c0500 ffff8e040d826780 ffff8e0430529680 CHR  /dev/pts/2

There's a clear outlier here, HTTPd is not normally run with sudo privileges, especially if it's running from an interactive terminal.

Before the kernel panic, a suspicious process was running with sudo privileges, What was the process name?

httpd-worker

crash> vm -p 8925
PID: 8925     TASK: ffff8e041046b100  CPU: 0    COMMAND: "httpd-worker"
       MM               PGD          RSS    TOTAL_VM
ffff8e040dc60cc0  ffff8e045695a000  1928k    17028k
      VMA           START       END     FLAGS FILE
ffff8e04104b9750 55b8c4756000 55b8c4758000 8000071 /usr/sbin/httpd-worker

IoCs

Execution indicators

  • /usr/sbin/httpd-worker
    • Called via bash > sudo > sudo
    • From an interactive terminal
  • /proc/jiffies_ext
    • Read via cat
    • From an interactive terminal

Kernel indicators

  • core_helper (Out-of-tree kernel module)
    • core_helper_read
    • core_helper_exit
    • proc_file_ops
    • cleanup_module

MITRE ATT&CK

← BACK