CUCM Trace Types & Prerequisites for Trace Collection
Trace Types: There are 2 kinds of trace to collect for CUCM troubleshooting; SDI (System Diagnostic Interface) and SDL (Signal Distribution Layer).
SDI Trace: mostly used by general voice engineers for troubleshooting the majority of CUCM issues. Plus, it is easier to understand than SDL.
SDL Trace: consists of large amount of low-level information (interactions of different state machines), normally used by Cisco TAC engineer to deeply investigate complex issues.
Prerequisites for Trace & Logs Collection:
- Enterprise Parameters -> Cisco Syslog Agent:
Remote Syslog Server Name: 10.10.10.10
Syslog Severity For Remote Syslog messages: Alert / Critical / Error / Warning / Notice
- Enterprise Parameters -> Max Number of Device Level Trace: 12 (default)
How many devices can be traced concurrently for device name-based trace
- Enterprise Parameters -> Trace Compression: Enable (Default: Disable)【not confirm】
- Enterprise Parameters For Log Compress: Begin with CUCM 8.0, Log compression is not configurable, and only CTI/CCM SDL&SDI traces are compressed. Compressed files (.gz extension) / compressed files being actively written (.gzo extension)
- Cisco RIS Data Collector Service -> Service Parameter:
TLC Throttling CPU Goal: 80 (up to 80% CPU for TLC)
TLC Throttling IOWait Goal: 10 (up to 10% IO for TLC)
- CallManager Service Parameter -> SDL Trace:
SDL Trace Flag: True (default)
Prerequisites for SIP Session Trace:
- Enterprise Parameters -> Call Trace Log Configuration for Session Trace:
Enable Call Trace Log: Enabled
Max Number of Call Trace Log Files: 2000 (default)
Call Trace Log File Size(MB): 2 (default)
Trace files are written in a round-robin fashion. When reaching the limit, CUCM will begin with the early trace file and overwrite it.
- CallManager Service Parameters -> Log Call-Related REFER/NOTIFY/SUBSCRIBE SIP Messages for Session Trace: Enabled.
Turn On/Off SDI&SDL trace & Trace level:
- Login Cisco Unified Serviceability Administration
- Navigate to Trace -> Configuration, select related CUCM node/Service, set SDI Trace on, Specify functional component to trace and trace information level (Detailed).
- As for SDL, click Go on the upper right corner which will lead you to configuration page for SDL trace, set SDL Trace on and also specify the information level.
- Go to Trace -> Troubleshooting Trace settings; globally switch on Trace for that CUCM Service.
Debug Trace Level Field: for trace level setting, recommended ones are Detailed or Arbitrary
- Detailed: provides detailed debug information and highly repetitive messages that are primarily used for debugging, including KeepAlives and Response.
- Arbitrary: provide low-level debug traces especially for difficult problems troubleshooting. It includes nearly everything in Detailed level except KeepAlives.
- Special: traces for all informational, non-repetitive messages such as process start-up messages, registration messages as well as all system and device initialization messages.
- State Transition: trace for call processing events in signaling layers
Trace & Log Central on RTMT
On RTMT, navigate to System -> Tools -> Trace -> Trace & Log Central;
- Remote Browse: view traces on the server without downloading the trace files
- Local Browse: Open local Trace & Log files using RTMT built-in viewers; Cisco Generic Viewer
and Cisco QRT Viewer.
- Query Wizard: Allows you to collect and download trace files that meets specific criteria that you specify, as well as to save that trace collection criteria for reuse.
- Collect Files: used to collect Trace & Log for specific services on CUCM.
- Schedule Collection: quite straight forward as the name suggests.
Here, for Trace & Log file collection purpose, Go to “Collect Files” and follow the instructions…
Real-Time Trace Viewing from CUCM CLI
On workstation where RTMT is not installed, CUCM CLI could be used to view newly-generated trace in real-time. Command “file tail …”focus on the tail of any file, so it will scroll down automatically to show you the lasted trace.
- Firstly, browse the file architecture of CUCM Server
admin: file list activelog /
dir count = 11, file count = 0
admin: file list activelog /cm/
dir count = 7, file count = 0
- Secondly, locate the latest trace file
admin: file list activelog /cm/trace/ccm/sdi/
dir count = 0, file count = 26
- Then, focus on the tail of the latest trace file
admin: file tail activelog /cm/trace/ccm/sdi/ccm00000025.txt
03/01/2002 05:51:12.373 CCM|CMProcMon – ——Entered Router Verification|<CLID::StandAloneCluster><NID::184.108.40.206><LVL::Arbitrary><MASK::0800>
03/01/2002 05:51:12.373 CCM|CMProcMon – —-Exited Router Verification|<CLID::StandAloneCluster><NID::220.127.116.11><LVL::Arbitrary><MASK::0800>
- Ctrl+C to exist the real-time viewing.
Useful Reference :
CUCM Trace Lookup for Different Scenarios: tell you which traces are required for a specific problem
MultiService Voice Debug Lookup: output related DEBUG commands for specific voice problem on Cisco IOS Platform
CUCM RTMT Trace Locations in CLIs: tells how to locate Different CUCM Traces under CLI.