Low-Level Diagnostics, Fault Management in Embedded Systems: Design, Implementation, Lessons Learned

Abstract

This article presents a practical perspective on designing and implementing fault management in embedded systems, particularly within automotive and industrial control contexts. It discusses the architecture of a layered fault-handling approach—comprising the Application Layer, Diagnostic Event Management (DEM) Wrapper, Low-Level Software (LLSW) Logic, and Hardware Monitoring. Through real-world observations and project-based experience, the article highlights challenges in fault detection, such as debounce logic, timing sensitivity, and noisy analog readings. Recommendations and lessons are drawn from specific case studies, including fault suppression through retry filters and handling sensor inaccuracies using median filters. The objective is to provide system engineers with actionable strategies for building fault-resilient embedded applications.

1. Introduction

Fault handling is a critical component in the reliability and safety of embedded systems. In safety-critical domains such as automotive, aerospace, and industrial automation, the ability to detect, log, and respond to faults in a timely and traceable manner can be the difference between product stability and costly field failures.

This article discusses an architecture that we have used in production systems, built on four conceptual layers:

• Application Layer
• DEM Wrapper
• LLSW Fault Log
• Hardware Monitoring Layer

While these components are common across many embedded frameworks (including AUTOSAR-based systems), their practical implementation requires careful balancing of timing constraints, hardware limitations, and system-level behavior. Drawing on lessons from real-world field issues and internal testing, this article aims to bridge the gap between abstract fault frameworks and implementation reality.

Figure 1: Fault Handling Architecture Diagram

2. Architectural Layers of Fault Handling

2.1 Application Layer

The Application Layer interfaces with both runtime logic and diagnostic components. Its primary role in fault management is to:

• Query fault states using Get_Fault() APIs
• Trigger test simulations using Set_Fault() APIs during validation
• Modify system behavior in response to active faults (e.g., initiate safe mode)

In one case involving a drivetrain control module, a recurring early-morning fault warning was traced back to the application layer correctly reporting a fault that was falsely triggered in LLSW. This highlighted the importance of validating the logic behind fault generation—not just the retrieval.

Design Insight: Application code should treat diagnostic queries as "truth," but diagnostic inputs must be traceable and rigorously validated.

2.2 DEM Wrapper

The DEM Wrapper abstracts the logging and state management of faults. Key functionalities include:

• Storing fault metadata (e.g., timestamps, frequency)
• Managing fault maturity and dematurity windows
• Providing consistent fault access APIs to the upper layers

For example, a heater fault would only be marked as “matured” if the fault condition persisted for a defined period—initially set to 50ms, later adjusted to 250ms after detecting false positives due to voltage sags during preconditioning.

Best Practice: Configure fault maturing/dematuring thresholds based on empirical testing. Excessively tight windows lead to noise; loose thresholds may suppress genuine failures.

2.3 LLSW Fault Management Logic

LLSW (Low-Level Software) is responsible for translating raw data into meaningful fault conditions. It typically:

• Applies retry logic (e.g., three consecutive failure samples before fault)
• Implements debounce filters to remove transient spikes
• Manages fault enablement based on runtime configuration

One project used a three-strike model with 10ms sampling intervals to determine a PWM signal drop. This was necessary to distinguish between true hardware failures and shutdown-induced signal tapering.

Tip: Use runtime flags to disable fault detection in transitional states (e.g., boot-up, self-test) to reduce false positives.

2.4 Hardware Monitoring Layer

At the lowest layer, the system interacts with physical components via sensors, ADCs, and digital inputs. This layer is where signal noise, EMI, and analog drift often undermine ideal fault detection.

An example involved a TI INA219 current sensor used for bus monitoring. EMI during high-power charging events caused voltage spikes, falsely triggering fault logic. A simple rolling median filter stabilized readings:

Lesson: Filtering at the hardware abstraction level can significantly improve fault signal integrity.

Figure 2: Sample Debounce Timing Flow

3. Case Study: Heater Fault During Preconditioning

Scenario: Fault was reported daily at 6:10 AM during scheduled battery preconditioning. The DEM logged an overcurrent event, but all signals normalized within seconds.

Diagnosis:

• Voltage dip during cold-start heater ramp-up
• LLSW triggered fault after a single sample (no retry)
• DEM maturing window too short (50ms)

Resolution:

• Implemented three-sample retry with 10ms spacing
• Extended maturing window to 250ms
• Added conditional skip for initial startup phase

Outcome: Fault no longer triggered falsely; system behavior stabilized without disabling the check entirely.

4. Recommendations

5. Conclusion

Fault management in embedded systems is rarely plug-and-play. While architectural patterns like DEM-LLSW separation are valuable, their real-world application requires careful tuning, environmental validation, and resilience against noisy hardware signals.

This article outlined a layered approach supported by field-tested observations and practical code snippets. A robust fault management design balances responsiveness, stability, and traceability—qualities essential for any safety-critical embedded system.