Cisco CUCM Trace Collection using RTMT

CUCM Trace Types & Prerequisites for Trace Collection
Trace Types
: There are 2 kinds of trace to collect for CUCM troubleshooting; SDI (System Diagnostic Interface) and SDL (Signal Distribution Layer).
SDI Trace: mostly used by general voice engineers for troubleshooting the majority of CUCM issues. Plus, it is easier to understand than SDL.
SDL Trace: consists of large amount of low-level information (interactions of different state machines), normally used by Cisco TAC engineer to deeply investigate complex issues.

Prerequisites for Trace & Logs Collection:

  • Enterprise Parameters -> Cisco Syslog Agent:
    Remote Syslog Server Name: 10.10.10.10
    Syslog Severity For Remote Syslog messages: Alert / Critical / Error / Warning / Notice
  • Enterprise Parameters -> Max Number of Device Level Trace: 12 (default)
    How many devices can be traced concurrently for device name-based trace
  • Enterprise Parameters -> Trace Compression: Enable (Default: Disable)【not confirm】
  • Enterprise Parameters For Log Compress: Begin with CUCM 8.0, Log compression is not configurable, and only CTI/CCM SDL&SDI traces are compressed. Compressed files (.gz extension) / compressed files being actively written (.gzo extension)
  • Cisco RIS Data Collector Service -> Service Parameter:
    TLC Throttling CPU Goal: 80 (up to 80% CPU for TLC)
    TLC Throttling IOWait Goal: 10 (up to 10% IO for TLC)
  • CallManager Service Parameter -> SDL Trace:
    SDL Trace Flag: True (default)

Prerequisites for SIP Session Trace:

  • Enterprise Parameters -> Call Trace Log Configuration for Session Trace:
    Enable Call Trace Log: Enabled
    Max Number of Call Trace Log Files: 2000 (default)
    Call Trace Log File Size(MB): 2 (default)
    Trace files are written in a round-robin fashion. When reaching the limit, CUCM will begin with the early trace file and overwrite it.
  • CallManager Service Parameters -> Log Call-Related REFER/NOTIFY/SUBSCRIBE SIP Messages for Session Trace: Enabled.

Turn On/Off SDI&SDL trace & Trace level:

  • Login Cisco Unified Serviceability Administration
  • Navigate to Trace -> Configuration, select related CUCM node/Service, set SDI Trace on, Specify functional component to trace and trace information level (Detailed).
  • As for SDL, click Go on the upper right corner which will lead you to configuration page for SDL trace, set SDL Trace on and also specify the information level.
  • Go to Trace -> Troubleshooting Trace settings; globally switch on Trace for that CUCM Service.

Debug Trace Level Field: for trace level setting, recommended ones are Detailed or Arbitrary

  • Detailed: provides detailed debug information and highly repetitive messages that are primarily used for debugging, including KeepAlives and Response.
  • Arbitrary: provide low-level debug traces especially for difficult problems troubleshooting. It includes nearly everything in Detailed level except KeepAlives.
  • Special: traces for all informational, non-repetitive messages such as process start-up messages, registration messages as well as all system and device initialization messages.
  • State Transition: trace for call processing events in signaling layers

Trace & Log Central on RTMT
On RTMT, navigate to System -> Tools -> Trace -> Trace & Log Central;

  • Remote Browse: view traces on the server without downloading the trace files
  • Local Browse: Open local Trace & Log files using RTMT built-in viewers; Cisco Generic Viewer
    and Cisco QRT Viewer.
  • Query Wizard: Allows you to collect and download trace files that meets specific criteria that you specify, as well as to save that trace collection criteria for reuse.
  • Collect Files: used to collect Trace & Log for specific services on CUCM.
  • Schedule Collection: quite straight forward as the name suggests.
    Here, for Trace & Log file collection purpose, Go to “Collect Files” and follow the instructions…

Real-Time Trace Viewing from CUCM CLI

On workstation where RTMT is not installed, CUCM CLI could be used to view newly-generated trace in real-time. Command “file tail …”focus on the tail of any file, so it will scroll down automatically to show you the lasted trace.

IPExpert blog-Trace File Viewing Made Easy in UCM

  • Firstly, browse the file architecture of CUCM Server
    admin: file list activelog /
    <dir>   car_db
    <dir>   ccm_db
    <dir>   cm
    <dir>   core
    <dir>   mgetty
    <dir>   mohprep
    <dir>   patches
    <dir>   platform
    <dir>   sa
    <dir>   syslog
    <dir>   tomcat
    dir count = 11, file count = 0
    admin: file list activelog /cm/
    <dir>   bin
    <dir>   cdr
    <dir>   cdr_repository
    <dir>   log
    <dir>   report
    <dir>   tftpdata
    <dir>   trace
    dir count = 7, file count = 0
    admin:
  • Secondly, locate the latest trace file
    admin: file list activelog /cm/trace/ccm/sdi/
    ccm00000001.txt                         ccm00000002.txt
    ccm00000003.txt                         ccm00000004.txt
    ccm00000005.txt                         ccm00000006.txt
    ccm00000007.txt                         ccm00000008.txt
    ccm00000009.txt                         ccm00000010.txt
    ccm00000011.txt                         ccm00000012.txt
    ccm00000013.txt                         ccm00000014.txt
    ccm00000015.txt                         ccm00000016.txt
    ccm00000017.txt                         ccm00000018.txt
    ccm00000019.txt                         ccm00000020.txt
    ccm00000021.txt                         ccm00000022.txt
    ccm00000023.txt                         ccm00000024.txt
    ccm00000025.txt                         ccm~num.bin
    dir count = 0, file count = 26
  • Then, focus on the tail of the latest trace file
    admin: file tail activelog /cm/trace/ccm/sdi/ccm00000025.txt
    03/01/2002 05:51:12.373 CCM|CMProcMon – ——Entered Router Verification|<CLID::StandAloneCluster><NID::142.100.64.11><LVL::Arbitrary><MASK::0800>
    03/01/2002 05:51:12.373 CCM|CMProcMon – —-Exited Router Verification|<CLID::StandAloneCluster><NID::142.100.64.11><LVL::Arbitrary><MASK::0800>
  • Ctrl+C to exist the real-time viewing.

Useful Reference :
CUCM Trace Lookup for Different Scenarios: tell you which traces are required for a specific problem
MultiService Voice Debug Lookup: output related DEBUG commands for specific voice problem on Cisco IOS Platform
CUCM RTMT Trace Locations in CLIs: tells how to locate Different CUCM Traces under CLI.