Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facing kernel panic for openEuler 23.09 #100

Closed
Sadiq-techie opened this issue Nov 8, 2023 · 19 comments
Closed

Facing kernel panic for openEuler 23.09 #100

Sadiq-techie opened this issue Nov 8, 2023 · 19 comments

Comments

@Sadiq-techie
Copy link

Hi Dong Du,

Facing kernel panic for openEuler 23.09

Workstation details:
Ubuntu 22.04
RISC-V cross compilier 13.2.0
qemu version 8.1.50

Steps to reproduce:
Cross compilied u-boot, opensbi-1.2 as per the steps for openEuler >23.
when cross compiling sdk it was unsuccessful.
used the https://ipads.se.sjtu.edu.cn:1313/d/6a464e02cd3d4c1bafb0/ sdk source and cross compilation was successful.

Booted the openEuler 23.09 with below command

qemu-system-riscv64 -nographic -machine virt \
			-smp 4 -m 2G \
			-bios  ./opensbi-1.2/build-oe/qemu-virt/platform/generic/firmware/fw_payload.bin  \
			-drive file=openEuler-23.09-qemu-riscv64.qcow2,format=qcow2,id=hd0 \
			-object rng-random,filename=/dev/urandom,id=rng0 \
			-device virtio-rng-device,rng=rng0 \
			-device virtio-blk-device,drive=hd0  \
			-device virtio-net-device,netdev=usernet \
			-netdev user,id=usernet,hostfwd=tcp::12055-:22 \
			-device qemu-xhci -usb -device usb-kbd -device usb-tablet

copied files from host to openEuler qemu as follows:

scp -P 12055 penglai-enclave-driver/penglai.ko root@localhost:~/
scp -P 12055 sdk/demo/host/host root@localhost:~/
scp -P 12055 sdk/demo/prime/prime root@localhost:~/

cross compiled penglai-kernel-driver generated penglai.ko
insert the enclave driver module
insmod penglai.ko

[root@openeuler-riscv64 penglai-enclave-driver]# insmod penglai.ko 
[ 1000.778139] penglai: loading out-of-tree module taints kernel.
[ 1000.780758] enclave_ioctl_init...
[Penglai KModule] sbi_ecall_penglai_host_handler invoked,funcid=100d
[Penglai Monitor] sm_mm_init invoked
[Penglai Monitor] sm_mm_init paddr:0xa3c00000, size:0x400000
[Debug:SM@dump_pmps] pmp_0: mode(0x18) perm(0x0) paddr(0x80028000) size(0x2000)
[Debug:SM@dump_pmps] pmp_1: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_2: mode(0x18) perm(0x7) paddr(0x0) size(0x8)
[Debug:SM@dump_pmps] pmp_3: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_4: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_5: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_6: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_7: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_8: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_9: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_10: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_11: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_12: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_13: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_14: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_15: mode(0x18) perm(0x7) paddr(0x0) size(0x8)
[Debug:SM@dump_pmps] pmp_0: mode(0x18) perm(0x0) paddr(0x80028000) size(0x2000)
[Debug:SM@dump_pmps] pmp_1: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_2: mode(0x18) perm(0x0) paddr(0xa3c00000) size(0x400000)
[Debug:SM@dump_pmps] pmp_3: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_4: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_5: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_6: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_7: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_8: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_9: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_10: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_11: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_12: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_13: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_14: mode(0x0) perm(0x0) paddr(0x0) size(0x0)
[Debug:SM@dump_pmps] pmp_15: mode(0x18) perm(0x7) paddr(0x0) size(0x8)
[Penglai Monitor] sm_mm_init ret:0d 
[ 1000.794471] [Penglai KModule] register_chrdev succeeded!

prime enclave execution was successful
./host prime
there is kernel call trace after enclave execution and kernel panic after some seconds

[root@openeuler ~]# rmo[  157.717647][  T730] kmemleak: Cannot insert 0xffffffd808391d80 into the object search tree (overlaps existing)
[  157.718217][  T730] CPU: 1 PID: 730 Comm: bash Tainted: G           OE      6.4.0-10.1.0.20.oe2309.riscv64 #1
[  157.718620][  T730] Hardware name: riscv-virtio,qemu (DT)
[  157.718938][  T730] Call Trace:
[  157.719111][  T730] [<ffffffff80006202>] dump_backtrace+0x28/0x30
[  157.719465][  T730] [<ffffffff80b96734>] show_stack+0x38/0x44
[  157.719699][  T730] [<ffffffff80ba9fe4>] dump_stack_lvl+0x44/0x5c
[  157.719946][  T730] [<ffffffff80baa014>] dump_stack+0x18/0x20
[  157.720172][  T730] [<ffffffff802a7ca8>] __create_object+0x374/0x398
[  157.720425][  T730] [<ffffffff80bacbbc>] kmemleak_alloc+0x48/0x80
[  157.720664][  T730] [<ffffffff80282e16>] __kmem_cache_alloc_node+0x1ca/0x2f8
[  157.720938][  T730] [<ffffffff802131e8>] __kmalloc+0x48/0x1b8
[  157.721162][  T730] [<ffffffff8036e65c>] ext4_htree_store_dirent+0x4c/0x16e
[  157.721432][  T730] [<ffffffff803a19c4>] htree_dirblock_to_tree+0x15c/0x27a
[  157.721694][  T730] [<ffffffff803a2836>] ext4_htree_fill_tree+0x172/0x2e6
[  157.721968][  T730] [<ffffffff8036dd08>] ext4_dx_readdir+0xd2/0x2be
[  157.722240][  T730] [<ffffffff8036e556>] ext4_readdir+0x43e/0x4c8
[  157.722505][  T730] [<ffffffff802d30a6>] iterate_dir+0x124/0x150
[  157.722772][  T730] [<ffffffff802d3330>] __do_sys_getdents64+0x64/0x150
[  157.723059][  T730] [<ffffffff802d3438>] sys_getdents64+0x1c/0x24
[  157.723342][  T730] [<ffffffff80baab12>] do_trap_ecall_u+0xf0/0x104
[  157.723653][  T730] [<ffffffff80003e70>] ret_from_exception+0x0/0x64
[  157.724090][  T730] kmemleak: Kernel memory leak detector disabled
[  157.724386][  T730] kmemleak: Object 0xffffffd808391d80 (size 64):
[  157.724653][  T730] kmemleak:   comm "bash", pid 730, jiffies 4294931765
[  157.724938][  T730] kmemleak:   min_count = 1
[  157.725136][  T730] kmemleak:   count = 0
[  157.725314][  T730] kmemleak:   flags = 0x1
[  157.725482][  T730] kmemleak:   checksum = 0
[  157.725654][  T730] kmemleak:   backtrace:
[  157.725902][  T730]  kmemleak_alloc+0x48/0x80
[  157.726108][  T730]  __kmem_cache_alloc_node+0x1ca/0x2f8
[  157.726322][  T730]  __kmalloc+0x48/0x1b8
[  157.726487][  T730]  ext4_htree_store_dirent+0x4c/0x16e
[  157.726693][  T730]  htree_dirblock_to_tree+0x15c/0x27a
[  157.726898][  T730]  ext4_htree_fill_tree+0x172/0x2e6
[  157.727098][  T730]  ext4_dx_readdir+0xd2/0x2be
[  157.727279][  T730]  ext4_readdir+0x43e/0x4c8
[  157.727455][  T730]  iterate_dir+0x124/0x150
[  157.727630][  T730]  __do_sys_getdents64+0x64/0x150
[  157.727826][  T730]  sys_getdents64+0x1c/0x24
[  157.728004][  T730]  do_trap_ecall_u+0xf0/0x104
[  157.728187][  T730]  ret_from_exception+0x0/0x64
[  157.738800][   T78] kmemleak: Automatic memory scanning thread ended

Similarly for 23.03 followed same steps with replacing openeuler 23.09 with 23.03 and was facing kernel panic as follows:

[ 5190.432458] ------------[ cut here ]------------
[ 5190.432866] WARNING: CPU: 3 PID: 1 at mm/slab_common.c:923 free_large_kmalloc+0x5a/0x90
[ 5190.433355] Modules linked in: drm fuse i2c_core drm_panel_orientation_quirks backlight [last unloaded: penglai(O)]
[ 5190.434170] CPU: 3 PID: 1 Comm: systemd Tainted: G           O       6.1.19-2.oe2303.riscv64 #1
[ 5190.434824] Hardware name: riscv-virtio,qemu (DT)
[ 5190.435223] epc : free_large_kmalloc+0x5a/0x90
[ 5190.435513]  ra : free_large_kmalloc+0x16/0x90
[ 5190.435762] epc : ffffffff8018e6fe ra : ffffffff8018e6ba sp : ff2000000060bb30
[ 5190.436112]  gp : ffffffff815db148 tp : ff60000001b10000 t0 : 0000000000000002
[ 5190.436451]  t1 : 0000000000000001 t2 : 0000000000000040 s0 : ff2000000060bb70
[ 5190.436808]  s1 : ff6000007ea00080 a0 : ff6000007ea00080 a1 : ff60000073636f72
[ 5190.437150]  a2 : ffffffffffffffff a3 : 0080000000000000 a4 : 0000000000000000
[ 5190.437540]  a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000000002
[ 5190.437904]  s2 : ff60000073636f72 s3 : ffffffff801830f4 s4 : 0000000000000800
[ 5190.438249]  s5 : ffffffff814c3d40 s6 : ffffffff815de228 s7 : 00000000ffffffff
[ 5190.438629]  s8 : ffffffff80000001 s9 : ff60000003a52410 s10: 0000000000000004
[ 5190.439059]  s11: ffffffff815df1d0 t3 : 0000000000000002 t4 : 0000000000000402
[ 5190.439450]  t5 : ff60000003a52430 t6 : ff6000000276afea
[ 5190.439740] status: 0000000200000120 badaddr: 0000000000000000 cause: 0000000000000003
[ 5190.440272] [<ffffffff8018e6fe>] free_large_kmalloc+0x5a/0x90
[ 5190.440694] [<ffffffff8018e844>] kfree+0x110/0x116
[ 5190.440965] [<ffffffff801830f4>] kfree_const+0x18/0x2e
[ 5190.441260] [<ffffffff80270336>] kernfs_put.part.0+0x6e/0x196
[ 5190.441550] [<ffffffff8027047c>] kernfs_put+0x1e/0x26
[ 5190.441820] [<ffffffff8026f694>] kernfs_evict_inode+0x2a/0x36
[ 5190.442126] [<ffffffff80217afa>] evict+0x94/0x160
[ 5190.442392] [<ffffffff80217f20>] iput+0x142/0x1bc
[ 5190.442687] [<ffffffff80212dfe>] dentry_unlink_inode+0xbe/0x108
[ 5190.442975] [<ffffffff802139fa>] __dentry_kill+0xb6/0x174
[ 5190.443176] [<ffffffff80215150>] shrink_dentry_list+0x4c/0xd6
[ 5190.443379] [<ffffffff8021533e>] shrink_dcache_parent+0xdc/0x12a
[ 5190.443634] [<ffffffff802068d0>] vfs_rmdir.part.0+0xc6/0x160
[ 5190.443954] [<ffffffff8020c564>] do_rmdir+0x170/0x180
[ 5190.444281] [<ffffffff8020c834>] sys_unlinkat+0x48/0x56
[ 5190.444528] [<ffffffff80003a7a>] ret_from_syscall+0x0/0x2
[ 5190.444904] ---[ end trace 0000000000000000 ]---
[ 5190.445400] object pointer: 0x000000003c5e4edc
[ 5190.445905] page:00000000c8b1a57b refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0xf3836
[ 5190.446401] flags: 0x0(zone=0)
[ 5190.447400] raw: 0000000000000000 ff6000007ea00088 ff6000007ea00088 0000000000000000
[ 5190.447880] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[ 5190.448619] page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0)
[ 5190.449097] ------------[ cut here ]------------
[ 5190.449359] kernel BUG at include/linux/mm.h:763!
[ 5190.449881] Kernel BUG [#1]
[ 5190.450119] Modules linked in: drm fuse i2c_core drm_panel_orientation_quirks backlight [last unloaded: penglai(O)]
[ 5190.450936] CPU: 3 PID: 1 Comm: systemd Tainted: G        W  O       6.1.19-2.oe2303.riscv64 #1
[ 5190.451455] Hardware name: riscv-virtio,qemu (DT)
[ 5190.451763] epc : __free_pages+0xcc/0xce
[ 5190.452049]  ra : __free_pages+0xcc/0xce
[ 5190.452296] epc : ffffffff801bb74a ra : ffffffff801bb74a sp : ff2000000060baf0
[ 5190.452699]  gp : ffffffff815db148 tp : ff60000001b10000 t0 : 6d75642065676170
[ 5190.453047]  t1 : 0000000000000070 t2 : 706d756420656761 s0 : ff2000000060bb30
[ 5190.453400]  s1 : ff6000007ea00080 a0 : 000000000000003e a1 : 0000000000000001
[ 5190.453824]  a2 : 0000000000000010 a3 : 0000000000000018 a4 : a811eb175061e400
[ 5190.454223]  a5 : a811eb175061e400 a6 : 0000000000000008 a7 : 0000000000000038
[ 5190.454649]  s2 : ff6000007ea00080 s3 : 0000000000000000 s4 : 0000000000000800
[ 5190.455058]  s5 : ffffffff814c3d40 s6 : ffffffff815de228 s7 : 00000000ffffffff
[ 5190.455471]  s8 : ffffffff80000001 s9 : ff60000003a52410 s10: 0000000000000004
[ 5190.455859]  s11: ffffffff815df1d0 t3 : ffffffff815f0717 t4 : ffffffff815f0717
[ 5190.456273]  t5 : ffffffff815f0718 t6 : ff2000000060b8c8
[ 5190.456541] status: 0000000200000120 badaddr: 0000000000000000 cause: 0000000000000003
[ 5190.456932] [<ffffffff801bb74a>] __free_pages+0xcc/0xce
[ 5190.457226] [<ffffffff8018e6f0>] free_large_kmalloc+0x4c/0x90
[ 5190.457581] [<ffffffff8018e844>] kfree+0x110/0x116
[ 5190.457871] [<ffffffff801830f4>] kfree_const+0x18/0x2e
[ 5190.458166] [<ffffffff80270336>] kernfs_put.part.0+0x6e/0x196
[ 5190.458502] [<ffffffff8027047c>] kernfs_put+0x1e/0x26
[ 5190.458838] [<ffffffff8026f694>] kernfs_evict_inode+0x2a/0x36
[ 5190.459132] [<ffffffff80217afa>] evict+0x94/0x160
[ 5190.459419] [<ffffffff80217f20>] iput+0x142/0x1bc
[ 5190.459708] [<ffffffff80212dfe>] dentry_unlink_inode+0xbe/0x108
[ 5190.460052] [<ffffffff802139fa>] __dentry_kill+0xb6/0x174
[ 5190.460358] [<ffffffff80215150>] shrink_dentry_list+0x4c/0xd6
[ 5190.460683] [<ffffffff8021533e>] shrink_dcache_parent+0xdc/0x12a
[ 5190.461019] [<ffffffff802068d0>] vfs_rmdir.part.0+0xc6/0x160
[ 5190.461345] [<ffffffff8020c564>] do_rmdir+0x170/0x180
[ 5190.461631] [<ffffffff8020c834>] sys_unlinkat+0x48/0x56
[ 5190.461927] [<ffffffff80003a7a>] ret_from_syscall+0x0/0x2
[ 5190.462686] ---[ end trace 0000000000000000 ]---
[ 5190.463124] note: systemd[1] exited with irqs disabled
[ 5190.463794] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 5190.464295] SMP: stopping secondary CPUs
[ 5190.465087] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

openEuler 23.03 uses 6.1.x Linux kernel and openEuler 23.09 uses 6.4.x Linux kernel.

observations:
when booted with openEuler 23.03 it is observed mmu was sv57 and with openEuler 23.09 it was sv39.
Any issue with OpenSBI?

Please let me know your suggestions or comments.

Thanks,
MD Sadiq

@Ddnirvana
Copy link
Collaborator

Hi Sadiq, thanks for your comments!

@fengerhu1 @Fly0307 Erhu and Xi, please help check the issue and help Sadiq to fix it out. Thanks!

@Fly0307
Copy link
Contributor

Fly0307 commented Nov 14, 2023

Hi Sadiq, We're sorry you're having this problem with penglai.
After our troubleshooting, the panic occurs because the kernel performs a scanning check on the memory, but it is unable to access the safe memory protected by penglai.
Specifically, in openEuler 23.03, this issue may cause intermittent triggering of kernel panics and eventual system termination. However, in openEuler 23.09, this issue most of the time only prompts a kernel scan message, but does not affect program and terminal execution, then you can choose to ignore the prompt.
We will fix this problem as soon as possible! If you need to use penglai, you can try openEuler 22.X (linux kernel is 5.X version) or lower version of openEuler, as far as we know currently in linux kernel 5.X did not occur kernel panic problem.

Fly0307 added a commit to Fly0307/Penglai-Enclave-sPMP that referenced this issue Nov 17, 2023
Add a sbi call to reclaim memory and clear pmp settings when uninstalling the pontoon module; add the release operation for kmalloc data

Penglai-Enclave#100
@Fly0307
Copy link
Contributor

Fly0307 commented Nov 21, 2023

This is an issue we have fixed in the latest version.
fix kernel panic

@Sadiq-techie
Copy link
Author

Hi Dong Du, Fly0307.
Thanks for the response and fix.
Also there is issue while cross compiling sdk as follows:
I was using gcc toolchain version 13.2.0 and was attempting to cross compile sdk but not succeeded.

make[2]: Entering directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/sdk/lib/app'
riscv64-unknown-linux-gnu-gcc -Wall -shared -fPIC -Iinclude -c src/eret.S -o src/eret.o
riscv64-unknown-linux-gnu-gcc -Wall -shared -fPIC -Iinclude -c src/print.c -o src/print.o
src/print.c:92:6: warning: conflicting types for built-in function 'vprintf'; expected 'int(const char *, void *)' [-Wbuiltin-declaration-mismatch]
   92 | void vprintf(const char*s, va_list vl)
      |      ^~~~~~~
src/print.c:6:1: note: 'vprintf' is declared in header '<stdio.h>'
    5 | #include "print.h"
  +++ |+#include <stdio.h>
    6 | 
riscv64-unknown-elf-ar rcs ../libpenglai-enclave-eapp.a src/eret.o src/print.o
make[2]: riscv64-unknown-elf-ar: No such file or directory
make[2]: *** [Makefile:24: ../libpenglai-enclave-eapp.a] Error 127
make[2]: Leaving directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/sdk/lib/app'
make[1]: *** [Makefile:2: all] Error 2
make[1]: Leaving directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/sdk/lib'
make: *** [Makefile:3: all] Error 2

As per the above log should I use multilib toolchain?
When I used multilib toolchain version 12.2.0 facing below issue

Entering directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/demo/gm_test_enclaves'
riscv64-unknown-linux-gnu-ld -static -L/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib -L/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/musl/lib -L/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm -lpenglai-enclave-eapp -lc -o test_sm2 test_sm2.o /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/libpenglai-enclave-eapp.a /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/sm2.a /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/miracl.a /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/musl/lib/libc.a /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/libgcc.a -T /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/app.lds
riscv64-unknown-linux-gnu-ld: /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/sm2.a(Random.o):(.bss+0x0): multiple definition of `g_mem'; /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/sm2.a(SM2_sv.o):(.bss+0x0): first defined here
riscv64-unknown-linux-gnu-ld: /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/sm2.a(Random.o):(.bss+0x278): multiple definition of `g_mem_point'; /home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/lib/gm/sm2.a(SM2_sv.o):(.bss+0x278): first defined here
riscv64-unknown-linux-gnu-ld: warning: test_sm2 has a LOAD segment with RWX permissions
make[2]: *** [Makefile:55: test_sm2] Error 1
make[2]: Leaving directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/demo/gm_test_enclaves'
make[1]: *** [Makefile:9: all] Error 2
make[1]: Leaving directory '/home/mdsadiq/qemu-virt/Penglai-openEuler-23.09-qemu-virt/Penglai-Enclave-sPMP/sdk/demo'
make: *** [Makefile:4: all] Error 2 

Please let me know your suggestions or comments.

Thanks,
MD Sadiq

@Fly0307
Copy link
Contributor

Fly0307 commented Nov 23, 2023

For sdk compiling, we've provided a docker image that contains most of the toolchain for penglai, the toolchain doesn't need to support multilib. you can start the image with this command:
docker run -v $(pwd):/home/penglai/penglai-enclave -w /home/penglai/penglai-enclave --network=host --rm -it ddnirvana/penglai-enclave:v0.5 bash
The version information of the toolchain in the container is as follows:

Target: riscv64-unknown-linux-gnu
Configured with: /home/dd/ddPC5/home/dd/devlop/penglai-enclave/penglai-u-sdk/riscv-gnu-toolchain/riscv-gcc/configure --target=riscv64-unknown-linux-gnu --prefix=/home/dd/ddPC5/home/dd/devlop/penglai-enclave/penglai-u-sdk/toolchain --with-sysroot=/home/dd/ddPC5/home/dd/devlop/penglai-enclave/penglai-u-sdk/toolchain/sysroot --with-system-zlib --enable-shared --enable-tls --enable-languages=c,c++,fortran --disable-libmudflap --disable-libssp --disable-libquadmath --disable-nls --disable-bootstrap --src=/home/dd/ddPC5/home/dd/devlop/penglai-enclave/penglai-u-sdk/riscv-gnu-toolchain/riscv-gcc --enable-checking=yes --disable-multilib --with-abi=lp64d --with-arch=rv64imafdc
Thread model: posix
gcc version 7.2.0 (GCC)

You can compile the sdk by executing the following command in the penglai-enclave-sPMP directory:

./docker_cmd.sh docker
# In the docker image
cd sdk
PENGLAI_SDK=$(pwd) make -j8

@Sadiq-techie
Copy link
Author

Hi,
Thanks for the quick response.
Any way to cross compile sdk with custom toolchain without using docker.

Please let me know your comments.

Thanks,
MD Sadiq

@fengerhu1
Copy link
Collaborator

@Sadiq-techie
It seems that you need to use the same version of the cross-compiler as we have in the docker.
Also, we'll try to update the compiler and fix any issues with it.

Thanks,
Erhu Feng

@Sadiq-techie
Copy link
Author

Hi Erhu Feng,

I want to use the same version of cross-compilier to build sdk, driver and opensbi.

Thanks,
MD Sadiq

@Sadiq-techie
Copy link
Author

Sadiq-techie commented Dec 15, 2023

Hi Erhu Feng,
observed an issue while inserting penglai kernel module as follows

[root@openeuler-riscv64 ~]# insmod penglai-enclave-driver/penglai.ko 
[ 2627.267942][ T1170] enclave_ioctl_init...
pmp memory overlaps with existing pmp memory!
[ 2627.270694][ T1170] [Penglai KModule]: sbi call mm_init is failed
insmod: ERROR: could not insert module penglai-enclave-driver/penglai.ko: Operation not permitted

In 1 out of 4 iterations observed this issue, driver cleanup is not occurring correctly during the module unloading process as below

[root@openeuler-riscv64 ~]# rmmod penglai 
[ 2946.111615][ T1179] enclave_ioctl_exit...
[ 2946.112191][ T1179] KERNEL MODULE:  free secmem:paddr:122000000, vaddr:ffffffd8a2000000, order:22
[ 2946.112597][ T1179] KERNEL MODULE:  free secmem:paddr:121800000, vaddr:ffffffd8a1800000, order:22

Please let me know your comments.

Thanks,
MD Sadiq

@Fly0307
Copy link
Contributor

Fly0307 commented Dec 20, 2023

@Sadiq-techie Can you provide more information about the execution instructions and the output before and after?
I'm not quite sure if you're executing the instructions to install and uninstall the penglai module correctly. If you uninstalled the penglai module correctly, it should output the following message:
M mode: sm_memory_reclaim: There is no mem to reclaim
After executing the rmmod penglai command, it looks like there isn't this output. Please provide more information if it's convenient and we'll fix it as soon as possible! Thanks!

@Fly0307
Copy link
Contributor

Fly0307 commented Dec 21, 2023

Hi Erhu Feng, observed an issue while inserting penglai kernel module as follows

[root@openeuler-riscv64 ~]# insmod penglai-enclave-driver/penglai.ko 
[ 2627.267942][ T1170] enclave_ioctl_init...
pmp memory overlaps with existing pmp memory!
[ 2627.270694][ T1170] [Penglai KModule]: sbi call mm_init is failed
insmod: ERROR: could not insert module penglai-enclave-driver/penglai.ko: Operation not permitted

In 1 out of 4 iterations observed this issue, driver cleanup is not occurring correctly during the module unloading process as below

[root@openeuler-riscv64 ~]# rmmod penglai 
[ 2946.111615][ T1179] enclave_ioctl_exit...
[ 2946.112191][ T1179] KERNEL MODULE:  free secmem:paddr:122000000, vaddr:ffffffd8a2000000, order:22
[ 2946.112597][ T1179] KERNEL MODULE:  free secmem:paddr:121800000, vaddr:ffffffd8a1800000, order:22

Please let me know your comments.

Thanks, MD Sadiq

We have fixed this issue in the latest version, you can try it at the following commit
fix free memory

@Sadiq-techie
Copy link
Author

Hi Erhu Feng,

When I was attempting to verify the TEE with the latest sources I was facing below issue

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[  500.802261][ T1110] enclave_ioctl_init...
[  500.803857][ T1110] [Penglai KModule] register_chrdev succeeded!
[root@openeuler-riscv64 ~]# ./host prime
[  503.370486][ T1114] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:306
[  503.370870][ T1114] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1114, name: host
[  503.371067][ T1114] preempt_count: 1, expected: 0
[  503.371184][ T1114] CPU: 0 PID: 1114 Comm: host Tainted: G        W  OE      6.4.0-g0e93d276d93e #1
[  503.371391][ T1114] Hardware name: riscv-virtio,qemu (DT)
[  503.371517][ T1114] Call Trace:
[  503.371601][ T1114] [<ffffffff80006202>] dump_backtrace+0x28/0x30
[  503.371762][ T1114] [<ffffffff80b94ac4>] show_stack+0x38/0x44
[  503.371904][ T1114] [<ffffffff80ba853c>] dump_stack_lvl+0x44/0x5c
[  503.372051][ T1114] [<ffffffff80ba856c>] dump_stack+0x18/0x20
[  503.372188][ T1114] [<ffffffff8004c9f0>] __might_resched+0x112/0x120
[  503.372336][ T1114] [<ffffffff8004ca48>] __might_sleep+0x4a/0x72
[  503.372478][ T1114] [<ffffffff80287920>] __kmem_cache_alloc_node+0x24c/0x2f8
[  503.372641][ T1114] [<ffffffff80216a20>] kmalloc_trace+0x32/0xc4
[  503.372788][ T1114] [<ffffffff032ccde2>] create_enclave+0x46/0x46a [penglai]
[  503.373088][ T1114] [<ffffffff032cd922>] penglai_enclave_create+0x10e/0x6a0 [penglai]
[  503.373372][ T1114] [<ffffffff032cedce>] penglai_enclave_ioctl+0xa8/0x20a [penglai]
[  503.373654][ T1114] [<ffffffff802d784e>] sys_ioctl+0x94/0xae
[  503.373790][ T1114] [<ffffffff80ba906a>] do_trap_ecall_u+0xf0/0x104
[  503.373939][ T1114] [<ffffffff80003e70>] ret_from_exception+0x0/0x64
host:1: enclave run
[  503.413973][ T1114] Prime is running
[  503.414503][ T1114] entry_point: 0000000000001000;	untrusted_ptr: 0000001000000000; untrusted_size: 0000000000002000
[  503.414692][ T1114] 
[  503.418964][ T1114] [Penglai Driver@penglai_enclave_run] run returned successfully

Please let me know your comments.

Thanks,
MD Sadiq

@Sadiq-techie
Copy link
Author

Hi Erhu Feng,

After the above mentioned issue, now unable to boot the kernel and stuck with below issue

pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
pmp size should be 2^power!
pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode

Please let me know your comments.

Thanks,
MD Sadiq

@Fly0307
Copy link
Contributor

Fly0307 commented Jan 16, 2024

Hi Erhu Feng,

After the above mentioned issue, now unable to boot the kernel and stuck with below issue

pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
pmp size should be 2^power!
pmp size should be 2^power!
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode
Warning: PMP region with paddr: 0x80000000 and size: 0x43000, can't be protected by pmp registers in NAPOT mode

Please let me know your comments.

Thanks, MD Sadiq

You can fix this with the following patch:0001-fix-2-power-problem.patch

@Sadiq-techie
Copy link
Author

Thanks Erhu Feng for the quick response, will apply this patch and verify.

@Sadiq-techie
Copy link
Author

Hi Erhu Feng,

When I was attempting to verify the TEE with the latest sources I was facing below issue

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[  123.154288][ T1082] penglai: loading out-of-tree module taints kernel.
[  123.156127][ T1082] penglai: module verification failed: signature and/or required key missing - tainting kernel
[  123.169273][ T1082] enclave_ioctl_init...
[  123.173332][ T1082] [Penglai KModule] register_chrdev succeeded!
[root@openeuler-riscv64 ~]# ./host prime
[  258.738913][ T1202] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:306
[  258.739226][ T1202] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1202, name: host
[  258.739435][ T1202] preempt_count: 1, expected: 0
[  258.739706][ T1202] CPU: 3 PID: 1202 Comm: host Tainted: G           OE      6.4.0-g0e93d276d93e #1
[  258.739931][ T1202] Hardware name: riscv-virtio,qemu (DT)
[  258.740133][ T1202] Call Trace:
[  258.740323][ T1202] [<ffffffff80006202>] dump_backtrace+0x28/0x30
[  258.740853][ T1202] [<ffffffff80b94ac4>] show_stack+0x38/0x44
[  258.740995][ T1202] [<ffffffff80ba853c>] dump_stack_lvl+0x44/0x5c
[  258.741141][ T1202] [<ffffffff80ba856c>] dump_stack+0x18/0x20
[  258.741280][ T1202] [<ffffffff8004c9f0>] __might_resched+0x112/0x120
[  258.741539][ T1202] [<ffffffff8004ca48>] __might_sleep+0x4a/0x72
[  258.741764][ T1202] [<ffffffff80287920>] __kmem_cache_alloc_node+0x24c/0x2f8
[  258.741952][ T1202] [<ffffffff80216a20>] kmalloc_trace+0x32/0xc4
[  258.742117][ T1202] [<ffffffff03793de2>] create_enclave+0x46/0x46a [penglai]
[  258.742700][ T1202] [<ffffffff03794922>] penglai_enclave_create+0x10e/0x6a0 [penglai]
[  258.743021][ T1202] [<ffffffff03795dce>] penglai_enclave_ioctl+0xa8/0x20a [penglai]
[  258.743339][ T1202] [<ffffffff802d784e>] sys_ioctl+0x94/0xae
[  258.743523][ T1202] [<ffffffff80ba906a>] do_trap_ecall_u+0xf0/0x104
[  258.743692][ T1202] [<ffffffff80003e70>] ret_from_exception+0x0/0x64
host:1: enclave attest
0xA7, 0x7A, 0xFA, 0xE8,  0x05, 0x43, 0x6C, 0x43,  0x3C, 0x59, 0x67, 0x54,  0x89, 0xA2, 0xCD, 0x5F,  
0xD1, 0xB1, 0x12, 0x7F,  0x83, 0x9A, 0xD6, 0xF8,  0x9E, 0x67, 0x75, 0xA1,  0xD4, 0xB2, 0x30, 0xF1,  
0x50, 0x54, 0x34, 0xE5,  0x0E, 0xC0, 0x52, 0xCB,  0x19, 0xDB, 0xA2, 0x09,  0x34, 0x73, 0xCB, 0x6C,  
0x97, 0xBF, 0xE2, 0xB2,  0x77, 0x87, 0x57, 0x51,  0x46, 0x90, 0x1F, 0xEC,  0x48, 0x49, 0x5F, 0x43,  
host:1: enclave run
[  258.832961][ T1202] Prime is running
[  258.841051][ T1202] [Penglai Driver@penglai_enclave_run] run returned successfully

Please let me know your comments or suggestions.

Thanks,
MD Sadiq

@Sadiq-techie
Copy link
Author

Sadiq-techie commented Jan 18, 2024

Hi Erhu Feng, below are the observations

with the latest sources observed the kernel bug.

with this commit 9a5e2dc there is no kernel bug, demo prime is working as expected.
In 1 out of 3 iterations facing below issue

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[ 1184.907648][ T1264] enclave_ioctl_init...
pmp memory overlaps with existing pmp memory!
[ 1184.909448][ T1264] [Penglai KModule]: sbi call mm_init is failed
insmod: ERROR: could not insert module penglai.ko: Operation not permitted

After uninstalling the penglai module correctly it output the following message:

[ 1279.395467][ T1277] enclave_ioctl_exit...
`[ 1279.396234][ T1277] KERNEL MODULE:  free secmem:paddr:11a400000, vaddr:ffffffd89a400000, order:22

Instead of below message
M mode: sm_memory_reclaim: There is no mem to reclaim

Final output:

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[ 1266.564089][ T1267] enclave_ioctl_init...
[ 1266.567769][ T1267] [Penglai KModule] register_chrdev succeeded!
[root@openeuler-riscv64 ~]# ./host prime
[ 1272.364446][ T1272] [Penglai Driver@create_enclave] total_pages:512 order:9
[ 1272.374211][ T1272] [Penglai Driver@penglai_enclave_loadelf] elf_prog_addr:0x1000 elf_prog_size:0x1070, infile_addr:0x3fb03ee000
[ 1272.374287][ T1272] [Penglai Driver@penglai_enclave_loadelf] elf_prog_addr:0x3000 elf_prog_size:0x0, infile_addr:0x3fb03f0000
[ 1272.375006][ T1272] [Penglai Driver@penglai_enclave_create] untrusted_mem->addr:0xffffffd89cbe2000 untrusted_mem->size:0x2000
[ 1272.375775][ T1272] [PENGLAI Driver@acquire_big_lock] penglai_enclave_create get lock
[ 1272.376320][ T1272] [Penglai Driver@penglai_enclave_create] enclave_mem->paddr:0x11f000000, size:0x200000
[ 1272.416968][ T1272] [PENGLAI Driver@release_big_lock] penglai_enclave_create release lock
host:1: enclave attest
[ 1272.418394][ T1272] [PENGLAI Driver@acquire_big_lock] penglai_enclave_attest get lock
[ 1272.438914][ T1272] [PENGLAI Driver@release_big_lock] penglai_enclave_attest release lock
0xC9, 0xF7, 0xEF, 0xD3,  0xCE, 0x4F, 0x31, 0xF9,  0xB5, 0xB9, 0x29, 0xEA,  0x52, 0xC7, 0x5A, 0xAF,  
0x76, 0xD8, 0x90, 0x08,  0x24, 0x81, 0x5D, 0xC1,  0xE2, 0x7B, 0x4B, 0x3E,  0x58, 0x88, 0xC3, 0xDC,  
0xCE, 0xD7, 0xFD, 0xCD,  0x02, 0x63, 0x18, 0x38,  0xB3, 0x63, 0x39, 0x29,  0x75, 0x42, 0xBA, 0xC0,  
0x7D, 0xFA, 0xBC, 0xEF,  0x6B, 0xFD, 0x72, 0x07,  0x2C, 0x6E, 0x68, 0x2B,  0x2D, 0xF0, 0xEC, 0xA9,  
host:1: enclave run
[ 1272.441885][ T1272] [PENGLAI Driver@acquire_big_lock] penglai_enclave_run get lock
[ 1272.442334][ T1272] [PENGLAI Driver@release_big_lock] penglai_enclave_run release lock
[ 1272.442647][ T1272] [Penglai Driver@penglai_enclave_run] goto infinite run loop
[ 1272.443498][ T1272] Prime is running
[Penglai Monitor@exit_enclave] retval of enclave is 2002
[ 1272.450513][ T1272] [PENGLAI Driver@acquire_big_lock] penglai_enclave_run get lock
[ 1272.450975][ T1272] [Penglai Driver@penglai_enclave_run] run returned successfully
[ 1272.452802][ T1272] [PENGLAI Driver@release_big_lock] penglai_enclave_run release lock
[root@openeuler-riscv64 ~]# rmmod penglai 
[ 1279.395467][ T1277] enclave_ioctl_exit...
[ 1279.396234][ T1277] KERNEL MODULE:  free secmem:paddr:11a400000, vaddr:ffffffd89a400000, order:22

`
Please let me know your suggestions.

Thanks,
MD Sadiq

@Fly0307
Copy link
Contributor

Fly0307 commented Jan 26, 2024

Hi Erhu Feng, below are the observations

with the latest sources observed the kernel bug.

with this commit 9a5e2dc there is no kernel bug, demo prime is working as expected. In 1 out of 3 iterations facing below issue

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[ 1184.907648][ T1264] enclave_ioctl_init...
pmp memory overlaps with existing pmp memory!
[ 1184.909448][ T1264] [Penglai KModule]: sbi call mm_init is failed
insmod: ERROR: could not insert module penglai.ko: Operation not permitted

After uninstalling the penglai module correctly it output the following message:

[ 1279.395467][ T1277] enclave_ioctl_exit...
`[ 1279.396234][ T1277] KERNEL MODULE:  free secmem:paddr:11a400000, vaddr:ffffffd89a400000, order:22

Instead of below message M mode: sm_memory_reclaim: There is no mem to reclaim

You're right, when using the latest commit test and closing the debug output, it prints log like this:

[root@openeuler ~]# rmmod penglai 
[   66.969733][  T826] enclave_ioctl_exit...
M mode:Finish free and there is no mem to reclaim

@Fly0307
Copy link
Contributor

Fly0307 commented Jan 26, 2024

Hi Erhu Feng,

When I was attempting to verify the TEE with the latest sources I was facing below issue

[root@openeuler-riscv64 ~]# insmod penglai.ko 
[  123.154288][ T1082] penglai: loading out-of-tree module taints kernel.
[  123.156127][ T1082] penglai: module verification failed: signature and/or required key missing - tainting kernel
[  123.169273][ T1082] enclave_ioctl_init...
[  123.173332][ T1082] [Penglai KModule] register_chrdev succeeded!
[root@openeuler-riscv64 ~]# ./host prime
[  258.738913][ T1202] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:306
[  258.739226][ T1202] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1202, name: host
[  258.739435][ T1202] preempt_count: 1, expected: 0
[  258.739706][ T1202] CPU: 3 PID: 1202 Comm: host Tainted: G           OE      6.4.0-g0e93d276d93e #1
[  258.739931][ T1202] Hardware name: riscv-virtio,qemu (DT)
[  258.740133][ T1202] Call Trace:
[  258.740323][ T1202] [<ffffffff80006202>] dump_backtrace+0x28/0x30
[  258.740853][ T1202] [<ffffffff80b94ac4>] show_stack+0x38/0x44
[  258.740995][ T1202] [<ffffffff80ba853c>] dump_stack_lvl+0x44/0x5c
[  258.741141][ T1202] [<ffffffff80ba856c>] dump_stack+0x18/0x20
[  258.741280][ T1202] [<ffffffff8004c9f0>] __might_resched+0x112/0x120
[  258.741539][ T1202] [<ffffffff8004ca48>] __might_sleep+0x4a/0x72
[  258.741764][ T1202] [<ffffffff80287920>] __kmem_cache_alloc_node+0x24c/0x2f8
[  258.741952][ T1202] [<ffffffff80216a20>] kmalloc_trace+0x32/0xc4
[  258.742117][ T1202] [<ffffffff03793de2>] create_enclave+0x46/0x46a [penglai]
[  258.742700][ T1202] [<ffffffff03794922>] penglai_enclave_create+0x10e/0x6a0 [penglai]
[  258.743021][ T1202] [<ffffffff03795dce>] penglai_enclave_ioctl+0xa8/0x20a [penglai]
[  258.743339][ T1202] [<ffffffff802d784e>] sys_ioctl+0x94/0xae
[  258.743523][ T1202] [<ffffffff80ba906a>] do_trap_ecall_u+0xf0/0x104
[  258.743692][ T1202] [<ffffffff80003e70>] ret_from_exception+0x0/0x64
host:1: enclave attest
0xA7, 0x7A, 0xFA, 0xE8,  0x05, 0x43, 0x6C, 0x43,  0x3C, 0x59, 0x67, 0x54,  0x89, 0xA2, 0xCD, 0x5F,  
0xD1, 0xB1, 0x12, 0x7F,  0x83, 0x9A, 0xD6, 0xF8,  0x9E, 0x67, 0x75, 0xA1,  0xD4, 0xB2, 0x30, 0xF1,  
0x50, 0x54, 0x34, 0xE5,  0x0E, 0xC0, 0x52, 0xCB,  0x19, 0xDB, 0xA2, 0x09,  0x34, 0x73, 0xCB, 0x6C,  
0x97, 0xBF, 0xE2, 0xB2,  0x77, 0x87, 0x57, 0x51,  0x46, 0x90, 0x1F, 0xEC,  0x48, 0x49, 0x5F, 0x43,  
host:1: enclave run
[  258.832961][ T1202] Prime is running
[  258.841051][ T1202] [Penglai Driver@penglai_enclave_run] run returned successfully

Please let me know your comments or suggestions.

Thanks, MD Sadiq

This message does not affect the normal execution of the program, if necessary you can solve this problem with this patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants