Troubleshooting a dual kernel configuration
|
===========================================
|
|
This page is a troubleshooting guide enumerating known issues
|
with dual kernel Xenomai configurations.
|
|
[TIP]
|
If running any release from the Xenomai 2 series, or a Xenomai 3
|
release using the *Cobalt* real-time core, then you are using a dual
|
kernel configuration, and this document was meant for you. Xenomai 3
|
over the *Mercury* core stands for a single kernel configuration
|
instead, for which you can find specific
|
link:troubleshooting-a-single-kernel-configuration/[troubleshooting
|
information here].
|
|
== Kernel-related issues
|
|
[[kconf]]
|
=== Common kernel configuration issues
|
|
When configuring the Linux kernel, some options should be avoided.
|
|
CONFIG_CPU_FREQ:: This allows the CPU frequency to be modulated with
|
workload, but many CPUs change the TSC counting frequency also, which
|
makes it useless for accurate timing when the CPU clock can
|
change. Also some CPUs can take several milliseconds to ramp up to
|
full speed.
|
|
CONFIG_CPU_IDLE:: Allows the CPU to enter deep sleep states,
|
increasing the time it takes to get out of these sleep states, hence
|
the latency of an idle system. Also, on some CPU, entering these deep
|
sleep states causes the timers used by Xenomai to stop functioning.
|
|
CONFIG_KGDB:: This option should not be enabled, except with x86.
|
|
CONFIG_CONTEXT_TRACKING_FORCE:: This option which appeared in kernel
|
3.8 is forced off by I-pipe patches since 3.14 onward, as it is
|
incompatible with interrupt pipelining, and has no upside for regular
|
users. However, you have to manually disable it for older kernels when
|
present. Common effects observed with this feature enabled include
|
RCU-related kernel warnings during real-time activities, and
|
pathologically high latencies.
|
|
=== Kernel hangs after "Uncompressing Linux... done, booting the kernel."
|
|
This means that the kernel crashes before the console is enabled. You
|
should enable the +CONFIG_EARLY_PRINTK+ option. For some architectures
|
(x86, arm), enabling this option also requires passing the
|
+earlyprintk+ parameter on the kernel command line. See
|
'Documentation/kernel-parameters.txt' for possible values.
|
|
For the ARM architecture, you have to enable +CONFIG_DEBUG_KERNEL+ and
|
+CONFIG_DEBUG_LL+ in order to be able to enable +CONFIG_EARLY_PRINTK+.
|
|
=== Kernel OOPSes
|
|
Please make sure to check the <<kconf,"Kernel configuration">> section
|
first.
|
|
If nothing seems wrong there, try capturing the OOPS information using
|
a _serial console_ or _netconsole_, then post it to the
|
mailto:xenomai@xenomai.org[xenomai mailing list], along with the
|
kernel configuration file (aka `.config`) matching the kernel build.
|
|
=== Kernel boots but does not print any message
|
|
Your distribution may be configured to pass the +quiet+ option on the
|
kernel command line. In this case, the kernel does not print all the
|
log messages, however, they are still available using the +dmesg+
|
command.
|
|
[[kerror]]
|
=== Kernel log displays Xenomai or I-pipe error messages
|
|
[[no-timer]]
|
==== I-pipe: could not find timer for cpu #N
|
|
The most probable reason is that no hardware timer chip is available
|
for Xenomai timing operations.
|
|
Check that you did not enable some of the conflicting options listed
|
in the <<kconf,"Kernel configuration">> section.
|
|
With AMD x86_64 CPUs:: You will most likely also see the following
|
message:
|
--------------------------------------------
|
I-pipe: cannot use LAPIC as a tick device
|
I-pipe: disable C1E power state in your BIOS
|
--------------------------------------------
|
The interrupt pipeline outputs this message if C1E option is enabled
|
in the BIOS. To fix this issue, disable C1E support in the BIOS. In
|
some Award BIOS this option is located in the +Advanced BIOS
|
Features->+ menu (+AMD C1E Support+).
|
|
[WARNING]
|
Disabling the +AMD K8 Cool&Quiet+ feature in the BIOS will *NOT* solve
|
this problem.
|
|
With other CPU architectures:: The interrupt pipeline implementation
|
may lack a registration for a hardware timer available to Xenomai
|
timing operations (e.g. a call to +ipipe_timer_register()+).
|
|
If you are working on porting the interrupt pipeline to some ARM SoC,
|
you may want to have a look at this
|
link:porting-xenomai-to-a-new-arm-soc/#The_general_case[detailed
|
information].
|
|
[[SMI]]
|
==== SMI-enabled chipset found, but SMI workaround disabled
|
|
You may have an issue with System Management Interrupts on your x86
|
platform. You may want to look at
|
link:dealing-with-x86-smi-troubles/[this document].
|
|
=== Xenomai and Linux devices share the same IRQ vector
|
|
This x86-specific issue might still happen on legacy hardware with no
|
MSI support. See
|
link:what-if-xenomai-and-linux-devices-share-the-same-IRQ[this
|
article] from the Knowledge Base.
|
|
=== Kernel issues specific to the Xenomai 2.x series
|
|
==== system init failed, code -19
|
|
See <<no-timer, this entry>>.
|
|
==== Local APIC absent or disabled!
|
|
The Xenomai 2.x _nucleus_ issues this warning if the kernel
|
configuration enables the local APIC support
|
(+CONFIG_X86_LOCAL_APIC+), but the processor status gathered at boot
|
time by the kernel says that no local APIC support is available.
|
There are two options for fixing this issue:
|
|
* either your CPU really has _no_ local APIC hardware, in which case
|
you need to rebuild a kernel with LAPIC support disabled.
|
|
* or it does have a local APIC but the kernel boot parameters did not
|
specify to activate it using the _lapic_ option. The latter is
|
required since 2.6.9-rc4 for boxes which APIC hardware is disabled by
|
default by the BIOS. You may want to look at the file
|
'Documentation/kernel-parameters.txt' from the Linux source tree, for
|
more information about this parameter.
|
|
== Application-level issues
|
|
[[vsyscall]]
|
=== --enable-x86-sep needs NPTL and Linux 2.6.x or higher
|
or,
|
|
=== --enable-x86-vsyscall requires NPTL ...
|
|
This message may happen when starting a Xenomai 2.x or 3.x application
|
respectively. On the x86 architecture, the configure script option
|
mentioned allows Xenomai to use the _vsyscall_ mechanism for issuing
|
system calls, based on the most efficient method determined by the
|
kernel for the current system. This mechanism is only available from
|
NPTL-enabled glibc releases.
|
|
Turn off this feature for other libc flavours.
|
|
=== Cobalt core not enabled in kernel
|
|
As mentioned in the message, the target kernel is lacking Cobalt
|
support. See
|
link:installing-xenomai-3-x/#Installing_the_Cobalt_core[this document]
|
for detailed information about installing Cobalt.
|
|
=== binding failed: Function not implemented
|
|
Another symptom of the previous issue, i.e. the Cobalt core is not
|
enabled in the target kernel.
|
|
=== binding failed: Operation not permitted
|
|
This is the result of an attempt to run a Xenomai application as an
|
unprivileged user, which fails because invoking Xenomai services
|
requires +CAP_SYS_NICE+. However, you may allow a specific group of
|
users to access Xenomai services, by following the instructions on
|
link:running-a-Xenomai-application-as-a-regular-user[this page].
|
|
=== incompatible ABI revision level
|
|
Same as below:
|
|
=== ABI mismatch
|
|
The ABI concerned by this message is the system call binary interface
|
between the Xenomai libraries and the real-time kernel services it
|
invokes (e.g. +libcobalt+ and the Cobalt kernel with Xenomai
|
3.x). This ABI may evolve over time, only between major Xenomai
|
releases or testing candidate releases (i.e. -rc series) though. When
|
this happens, the ABI level required by the application linked against
|
Xenomai libraries may not match the ABI exposed by the Xenomai
|
co-kernel implementation on the target machine, which is the situation
|
this message reports.
|
|
To fix this issue, just make sure to rebuild both the Xenomai kernel
|
support and the user-space binaries for your target system. If however
|
you did install the appropriate Xenomai binaries on your target
|
system, chances are that stale files from a previous Xenomai
|
installation still exist on your system, causing the mismatch.
|
|
Each major Xenomai release (e.g. 2.1.x, 2.2.x ... 2.6.x, 3.0.x ...)
|
defines such kernel/user ABI, which remains stable across minor update
|
releases (e.g. 2.6.0 -> 2.6.4). This guarantee makes partial updates
|
possible with production systems (i.e. kernel and/or user support).
|
For instance, any application built over the Xenomai 2.6.0 binaries
|
can run over a Xenomai 2.6.4 kernel support, and conversely.
|
|
[TIP]
|
Debian-based distributions (notably Ubuntu) may ship with
|
pre-installed Xenomai libraries. Make sure that these files don't get
|
in the way if you plan to install a more recent Xenomai kernel
|
support.
|
|
=== <program>: not found
|
|
Although the program in question may be present, this message may
|
happen on ARM platforms when a mismatch exists between the kernel and
|
user library configurations with respect to EABI support. Typically,
|
if user libraries are compiled with a toolchain generating OABI code,
|
the result won't run over a kernel not enabling the
|
+CONFIG_OABI_COMPAT+ option. Conversely, the product of a compilation
|
with an EABI toolchain won't run on a kernel not enabling the
|
+CONFIG_AEABI+ option.
|
|
=== incompatible feature set
|
|
When a Xenomai application starts, the set of core features it
|
requires is compared to the feature set the kernel provides. This
|
message denotes a mismatch between both sets, which can be solved by
|
fixing the kernel and/or user build configuration. Further details
|
are available from link:installing-xenomai-3-x[this page] for Xenomai
|
3, and link:installing-xenomai-2-x[this page] for Xenomai 2.
|
|
==== feature mismatch: missing="smp/nosmp"
|
|
On SMP-capable architectures, both kernel and user-space components
|
(i.e. Xenomai libraries) must be compiled with the same setting with
|
respect to SMP support.
|
|
SMP support in the kernel is controlled via the +CONFIG_SMP+ option.
|
The +--enable-smp+ configuration switch enables this feature for the
|
Xenomai libraries (conversely, +--disable-smp+ disables it).
|
|
[CAUTION]
|
Using Xenomai libraries built for a single-processor configuration
|
(i.e. +--disable-smp+) over a SMP kernel (i.e. +CONFIG_SMP=y+) is
|
*NOT* valid. On the other hand, using Xenomai libraries built with SMP
|
support enabled over a single-processor kernel is fine.
|
|
=== Application-level issues specific to the Xenomai 2.x series
|
|
The following feature mismatches can be detected with the 2.x series:
|
|
==== feature mismatch: missing="kuser_tsc"
|
|
See the <<arm-tsc, "ARM tsc emulation issues">> section.
|
|
[NOTE]
|
This issue does not affect Xenomai 3.x as the latter requires modern
|
I-pipe series which must provide _KUSER_TSC_ support on the ARM
|
architecture.
|
|
==== feature mismatch: missing="sep"
|
|
This error is specific to the x86 architecture on Xenomai 2.x, for
|
pre-Pentium CPUs which do not provide the _sysenter/sysexit_
|
instruction pair. See <<vsyscall, this section>>.
|
|
[NOTE]
|
This issue does not affect Xenomai 3.x as the latter does not
|
support pre-Pentium systems in the first place.
|
|
==== feature mismatch: missing="tsc"
|
|
This error is specific to the x86 architecture on Xenomai 2.x, for
|
pre-Pentium CPUs which do not provide the _rdtsc_ instruction. In this
|
particular case, +--enable-x86-tsc+ cannot be mentioned in the
|
configuration options for building the user libraries, since the
|
processor does not support this feature.
|
|
The rule of thumb is to pick the *exact* processor for your x86
|
platform when configuring the kernel, at the very least the most
|
specific model which is close to the target CPU, not a generic
|
placeholder such as _i586_, for which _rdtsc_ is not available.
|
|
If your processor does not provide the _rdtsc_ instruction, you have
|
to pass +--disable-x86-tsc+ option to the configure script for
|
building the user librairies. In this case, Xenomai will provide a
|
(much slower) emulation of the hardware TSC.
|
|
[NOTE]
|
This issue does not affect Xenomai 3.x as the latter does not
|
support pre-Pentium systems in the first place.
|
|
[[arm-tsc]]
|
==== ARM tsc emulation issues
|
|
In order to allow applications to measure short durations with as
|
little overhead as possible, Xenomai uses a 64 bits high resolution
|
counter. On x86, the counter used for this purpose is the time-stamp
|
counter readable by the dedicated _rdtsc_ instruction.
|
|
ARM processors generally do not have a 64 bits high resolution counter
|
available in user-space, so this counter is emulated by reading
|
whatever high resolution counter is available on the processor, and
|
used as clock source in kernel-space, and extend it to 64 bits by
|
using data shared with the kernel. If Xenomai libraries are compiled
|
without emulated tsc support, system calls are used, which have a much
|
higher overhead than the emulated tsc code.
|
|
In recent versions of the I-pipe patch, SOCs generally select the
|
+CONFIG_IPIPE_ARM_KUSER_TSC+ option, which means that the code for
|
reading this counter is provided by the kernel at a predetermined
|
address (in the vector page, a page which is mapped at the same
|
address in every process) and is the code used if you do not pass the
|
+--enable-arm-tsc+ or +--disable-arm-tsc+ option to configure, or pass
|
+--enable-arm-tsc=kuser+.
|
|
This default should be fine with recent patches and most ARM
|
SOCs.
|
|
However, if you see the following message:
|
-------------------------------------------------------------------------------
|
incompatible feature set
|
(userland requires "kuser_tsc...", kernel provides..., missing="kuser_tsc")
|
-------------------------------------------------------------------------------
|
|
It means that you are either using an old patch, or that the SOC you
|
are using does not select the +CONFIG_IPIPE_ARM_KUSER_TSC+ option.
|
|
So you should resort to what Xenomai did before branch 2.6: select the
|
tsc emulation code when compiling Xenomai user-space support by using
|
the +--enable-arm-tsc+ option. The parameter passed to this option is
|
the name of the SOC or SOC family for which you are compiling Xenomai.
|
Typing:
|
-------------------------------------------------------------------------------
|
/patch/to/xenomai/configure --help
|
-------------------------------------------------------------------------------
|
|
will return the list of valid values for this option.
|
|
If after having enabled this option and recompiled, you see the
|
following message when starting the latency test:
|
-------------------------------------------------------------------------------
|
kernel/user tsc emulation mismatch
|
-------------------------------------------------------------------------------
|
or
|
-------------------------------------------------------------------------------
|
Hardware tsc is not a fast wrapping one
|
-------------------------------------------------------------------------------
|
|
It means that you selected the wrong SOC or SOC family, reconfigure
|
Xenomai user-space support by passing the right parameter to
|
+--enable-arm-tsc+ and recompile.
|
|
The following message:
|
-------------------------------------------------------------------------------
|
Your board/configuration does not allow tsc emulation
|
-------------------------------------------------------------------------------
|
|
means that the kernel-space support for the SOC you are using does not
|
provide support for tsc emulation in user-space. In that case, you
|
should recompile Xenomai user-space support passing the
|
+--disable-arm-tsc+ option.
|
|
==== hardware tsc is not a fast wrapping one
|
or,
|
|
==== kernel/user tsc emulation mismatch
|
or,
|
|
==== board/configuration does not allow tsc emulation
|
|
See the <<arm-tsc, "ARM tsc emulation issues">> section.
|
|
==== native skin or CONFIG_XENO_OPT_PERVASIVE disabled
|
|
Possible reasons for this error are:
|
|
* you booted a kernel without Xenomai or I-pipe support, a kernel with
|
I-pipe and Xenomai support should have a '/proc/ipipe/version' and
|
'/proc/xenomai/version' files;
|
|
* the kernel you booted does not have the +CONFIG_XENO_SKIN_NATIVE+ and
|
+CONFIG_XENO_OPT_PERVASIVE+ options enabled;
|
|
* Xenomai failed to start, check the <<kerror,"Xenomai or I-pipe error
|
in the kernel log">> section;
|
|
* you are trying to run Xenomai user-space support compiled for x86_32
|
on an x86_64 kernel.
|
|
==== "warning: <service> is deprecated" while compiling kernel code
|
|
Where <service> is a thread creation service, one of:
|
|
* +cre_tsk+
|
* +pthread_create+
|
* +rt_task_create+
|
* +sc_tecreate+ or +sc_tcreate+
|
* +taskSpawn+ or +taskInit+
|
* +t_create+
|
|
Starting with Xenomai 3, APIs are not usable from kernel modules
|
anymore, at the notable exception of the RTDM device driver API, which
|
by essence must be used from kernel space for writing real-time device
|
drivers. Those warnings are there to remind you that application code
|
should run in user-space context instead, so that it can be ported to
|
Xenomai 3.
|
|
You may switch those warnings off by enabling the
|
+CONFIG_XENO_OPT_NOWARN_DEPRECATED+ option in your kernel
|
configuration, but nevertheless, you have been *WARNED*.
|
|
==== a Xenomai system call fails with code -38 (ENOSYS)
|
|
Possible reasons for this error are:
|
|
* you booted a kernel without Xenomai or I-pipe support, a kernel with
|
I-pipe and Xenomai support should have a '/proc/ipipe/version' and
|
'/proc/xenomai/version' files;
|
|
* the kernel you booted does not have the +CONFIG_XENO_SKIN_*+ option
|
enabled for the skin you use, or +CONFIG_XENO_OPT_PERVASIVE+ is
|
disabled;
|
|
* Xenomai failed to start, check the <<kerror,"Xenomai or I-pipe error
|
in the kernel log">> section;
|
|
* you are trying to run Xenomai user-space support compiled for x86_32
|
on an x86_64 kernel.
|
|
==== the application overconsumes system memory
|
|
Your user-space application unexpectedly commits a lot of virtual
|
memory, as reported by "+top+" or '/proc/<pid>/maps'. Sometimes OOM
|
situations may even appear during runtime on systems with limited
|
memory.
|
|
The reason is that Xenomai threads are underlaid by regular POSIX
|
threads, for which a large default amount of stack space memory is
|
commonly reserved by the POSIX threading library (8MiB per thread by
|
the _glibc_). Therefore, the kernel will commit as much as
|
_8MiB{nbsp}*{nbsp}nr_threads_ bytes to RAM space for the application,
|
as a side-effect of calling the +mlockall()+ service to lock the
|
process memory, as Xenomai requires.
|
|
This behaviour can be controlled in two ways:
|
|
- via the _stacksize_ parameter passed to the various thread creation
|
routines, or +pthread_attr_setstacksize()+ directly when using the
|
POSIX API.
|
|
- by setting a lower user-limit for the initial stack allocation from
|
the application's parent shell which all threads from the child
|
process inherit, as illustrated below:
|
|
---------------------------------------------------------------------
|
ulimit -s <initial-size-in-kbytes>
|
---------------------------------------------------------------------
|
|
==== freeze or machine lockup
|
|
Possible reasons may be:
|
|
- Stack space overflow issue now biting some real-time kernel thread?
|
|
- Spurious delay/timeout values computed by the application
|
(specifically: too short).
|
|
- A case of freeze is a system call called in a loop which fails
|
without its return value being properly checked.
|
|
On x86, whenever the nucleus watchdog does not trigger, you may want to
|
try disabling CONFIG_X86_UP_IOAPIC while keeping CONFIG_X86_UP_APIC, and
|
arm the kernel NMI watchdog on the LAPIC (nmi_watchdog=2). You may be
|
lucky and have a backtrace after the freeze. Maybe enabling all the
|
nucleus debug options would catch something too.
|
|
== Issues when running Xenomai test programs
|
|
[[latency]]
|
=== Issues when running the _latency_ test
|
|
The first test to run to see if Xenomai is running correctly on your
|
platform is the latency test. The following sections describe the
|
usual reasons for this test not to run correctly.
|
|
==== failed to open benchmark device
|
|
You have launched +latency -t 1+ or +latency -t 2+ which both require
|
the kernel to have been configured with the
|
+CONFIG_XENO_DRIVERS_TIMERBENCH+ option.
|
|
==== the _latency_ test hangs
|
|
The most common reason for this issues is a too short period passed
|
with the +-p+ option, try increasing the period. If you enable the
|
watchdog (option +CONFIG_XENO_OPT_WATCHDOG+, in your kernel
|
configuration), you should see the <<short-period, "watchdog triggered
|
(period too short?)">> message.
|
|
[[short-period]]
|
==== watchdog triggered (period too short?)
|
|
The built-in Xenomai watchdog has stopped the _latency_ test because
|
it was using all the CPU in pure real-time mode (aka _primary
|
mode_). This is likely due to a too short period. Run the _latency_
|
test again, passing a longer period using the +-p+ option this time.
|
|
==== the _latency_ test shows high latencies
|
|
The _latency_ test runs, but you are seeing high latencies.
|
|
* make sure that you carefully followed the <<kconf,"Kernel
|
configuration" section>>.
|
|
* if running on a Raspberry Pi SBC, make sure you don't hit a firmware
|
issue, see https://github.com/raspberrypi/firmware/issues/497.
|
|
* if running on a x86 platform, make sure that you do not have an
|
issue with SMIs, see the <<SMI, section about SMIs>>.
|
|
* if running on a x86 platform with a _legacy USB_ switch available
|
from the BIOS configuration, try disabling it.
|
|
* if you do not have this option at BIOS configuration level, it does
|
not necessarily mean that there is no support for it, thus no
|
potential for high latencies; this support might just be forcibly
|
enabled at boot time. To solve this, in case your machine has some USB
|
controller hardware, make sure to enable the corresponding host
|
controller driver support in your kernel configuration. For instance,
|
UHCI-compliant hardware needs +CONFIG_USB_UHCI_HCD+. As part of its
|
init chores, the driver should reset the host controller properly,
|
kicking out the BIOS off the concerned hardware, and deactivate the
|
USB legacy mode if set in the same move.
|
|
* if you observe high latencies while running X-window, try disabling
|
hardware acceleration in the X-window server file. With recent
|
versions of X-window, try using the 'fbdev' driver. Install it
|
(Debian package named 'xserver-xorg-video-fbdev' for instance), then
|
modifiy the +Device+ section to use this driver in
|
'/etc/X11/xorg.conf', as in:
|
-------------------------------------------------------------------------------
|
Section "Device"
|
Identifier "Card0"
|
Driver "fbdev"
|
EndSection
|
-------------------------------------------------------------------------------
|
With olders versions of X-window, keep the existing driver, but
|
add the following line to the +Device+ section:
|
-------------------------------------------------------------------------------
|
Option "NoAccel"
|
-------------------------------------------------------------------------------
|
|
=== Issues when running the _switchtest_ program
|
|
==== pthread_create: Resource temporarily unavailable
|
|
The switchtest test creates many kernel threads, an operation which
|
consumes memory taken from internal pools managed by the Xenomai
|
real-time core.
|
|
Xenomai 2.x and 3.x series require +CONFIG_XENO_OPT_SYS_HEAPSZ+ to be
|
large enough in the kernel configuration settings, to cope with the
|
allocation requests.
|
|
Xenomai 2.x may also require to increase the
|
+CONFIG_XENO_OPT_SYS_STACKPOOLSZ+ setting.
|
