Red Hat asked you for an SOSreport, What now.

Hits: 159

SOSreport for Red Hat support

OK Guys – So Redhat support asked you for an SOSreport. Hopefully, they will look at it. Here is some of their instructions that are hard to find in google.

 

A devops pal of mine joked that he makes him mom send him an SOSreport whenever she has an issue, especially with those new programmable IOT ovens and refrigerators.

For Azure Virtual Guests: Follow the directions in: “How to Create a Microsoft Azure Virtual Machine sosreport and Submit the Report to Red Hat” for Azure virtual guests.

Resolution

What is a sosreport:

The sosreport command is a tool that collects configuration details, system information and diagnostic information from a Red Hat Enterprise Linux system. For instance: the running kernel version, loaded modules, and system and service configuration files. The command also runs external programs to collect further information, and stores this output in the resulting archive.


How to run sosreport:

Once the sos package has been installed, issue the following command to run sosreport:

I was asked to run:
# sosreport –batch –all-logs
# sosreport
  • Note: sosreport requires root permissions to run.
  • Hangs: sosreport can take a long time to run, but if the command is not completing/hung: follow the steps in What to do if sosreport hangs.
  • If you get “command not found” error, verify sos package has been installed and installation has no errors.

The command will normally complete within a few minutes on Red Hat Enterprise Linux 6 and 7.

Once completed, sosreport will generate a compressed a file under /tmp (for RHEL6 and earlier) or under /var/tmp (for RHEL7 and later). Different versions use different compression schemes (gzbz2, or xz). The file should be provided to your support representative (normally as an attachment to an open support case).

The size of the archive varies depending on system configuration and any optional sosreport features that are enabled (for example specifying the “all_logs” option of the general module to collect all syslog log files may greatly increase the size of the archive).

To post sosreport, or any other file, to a existing support case you can use the redhat-support-tool command line option, the Red Hat Portal UI or several different methods using FTP.

If the collected sosreport file is too big to upload to the case, it could be uploaded to the Red Hat ftp dropbox.redhat.com site.

To avoid the need to enter user and account information interactively the command may be run in batch mode by use of the --batch option. In this case user information is obtained from the system RHN configuration files:

# sosreport --batch

The sosreport command has a modular structure and allows the user to enable and disable modules and specify module options via the command line. To list available modules (plug-ins) use the following command:

# sosreport -l

To temporarily turn off a module:

Include it in a comma-separated list of modules passed to the -n/--skip-plugins option.

For instance, to temporarily disable both the kvm and amd modules (if broken):

# sosreport -n kvm,amd

Individual modules may provide additional options that may be specified via the -k option. For example on Red Hat Enterprise Linux 4 and 5 installations the sos rpm module collects “rpm -Va” output by default. As this may be time-consuming the behavior may be disabled via:

# sosreport -k rpm.rpmva=off

Running sosreport on RHN Proxy Server
To capture more detailed information on RHN Proxy Server, run the following command:

# sosreport -o rhn

Note:

1. Red Hat Enterprise Linux 4.5 and earlier include the older sysreport command for creating diagnostic archives. If possible update your system to the sos package included in later releases however sysreports may still be submitted to Red Hat if needed What is a sysreport and how to run it in Red Hat Enterprise Linux?

2. The sos package in Red Hat Enterprise Linux 4 and 5 also includes the sysreport command. This is a symbolic link provided for backwards compatibility:

# ls -l /usr/sbin/sysreport
lrwxrwxrwx 1 root root 19 Nov  3  2008 /usr/sbin/sysreport -> /usr/sbin/sosreport

To use the original (legacy) version of the sysreport script on these installations use the sysreport.legacy command:

# sysreport.legacy

This utility will go through and collect some detailed information about the hardware and setup of your Red Hat Linux system. This information will be used to diagnose problems with your system and will be considered confidential information. Red Hat will use this information for diagnostic purposes ONLY.


Installing sosreport:

The sos package must be installed in order to run the sosreport command. The package is part of the default group and will be installed automatically on most systems. You can check to see if the sos package is installed and whether there is any problems with the installation using the following command:

# rpm -qa | grep sos
sos-3.2-35.el7_2.3.noarch                  << sos package is installed
# rpm -V sos                               << run verification on installed package
#  

If for any reason the package is not present you will receive errors when attempting to run the command such as:

# sosreport
-bash: sosreport: command not found

Then the sos package, including the sosreport command, must be manually installed using the following steps:

  • Red Hat Enterprise Linux 5 and later
    • If the system is registered with RHSM, use the yum command:
      # yum install sos
      
    • If the system is not registered with RHSM, the sos package can be downloaded from the RHN website or found on the installation CDs or DVD. The rpm command may be used to install the package on any version of Red Hat Enterprise Linux:
      # rpm -Uvh sos-<version>.noarch.rpm
      

What to do if sosreport hangs:

  • RHEL5 Warning: If running on RHEL5, background the sosreport process (ctrl-z) before killing it via its PID. Otherwise you may encounter the issue documented in KCS#55275
  • Disk Space Issues: If sosreport fails due to “No space left on device
    • Verify there is enough space on the filesystem containing /tmp and free additional space if needed:
      # df -h /tmp
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/vg_ren-lv_root
                             50G   49G    1M  99% /
      
    • Or, Generate sosreport to an alternative location
  • Hangs: The sosreport may have hung because of a specific plugin
    • You can show all available sosreport plugin modules via:
      sosreport -l
      
    • Try to determine which plugin it is hanging on.
      • Increase verbosity sosreport -vvvv
        • If a plugin times out or last plugin does not finish, rerun the sosreport and skip that plugin:
          sosreport -v -n <plugin_name>
          
        • For example, to exclude the block and filesys plugin modules:
          sosreport -n block,filesys
          
      • strace it and once you determine which plugin it is hanging on, exclude it via above.
  • Other: Otherwise, instead of collecting the failure sosreport, a manual report may be created by running the following script. Note this will only be a portion of what is collected in a standard sosreport.
        rm -Rf /tmp/$(hostname)_hungsos 2> /dev/null
        mkdir /tmp/$(hostname)_hungsos
        cd   /tmp/$(hostname)_hungsos
        uname -a &> uname
        cat /proc/cmdline &> cmdline
        cat /proc/sys/kernel/tainted &> tainted
        chkconfig --list &> chkconfig
        date &> date
        df &> df
        dmesg &> dmesg
        dmidecode &> dmidecode
        lls -lahR /boot &> ls_lahR.boot 
        lls -lahR /dev &> ls_lahR.dev
        cat /proc/scsi/scsi &> proc_scsi_scsi
        free &> free
        sar -A &> sar.out
        hostname --fqdn &> hostname
        ifconfig &> ifconfig
        lsmod &> lsmod 
        lspci &> lspci
        cat /proc/mounts &> mount
        netstat -tlpn &> netstat
        ps auxww &> ps
        ulimit -a &> ulimit
        uname -a &> uname
        uptime &> uptime
        cat /proc/meminfo &> meminfo
        cat /proc/cpuinfo &> cpuinfo
        mkdir etc
        cd etc 
        cp /etc/fstab .
        cp /etc/cluster/cluster.conf . &> /dev/null
        cp /etc/security/limits.conf .
        cp /etc/redhat-release .
        cp /etc/sysctl.conf .
        cp /etc/multipath.conf . &> /dev/null
        cp /etc/modprobe.conf .  &>/dev/null
        mkdir modprobe
        cd   modprobe
        cp /etc/modprobe/* . -R &> /dev/null
        cd /tmp/$(hostname)_hungsos/etc
        mkdir sysconfig/network-scripts -p
        cd sysconfig
        cp /etc/sysconfig/* . -R
        cd /tmp/$(hostname)_hungsos
        mkdir var/log -p
        cp /var/log/message* var/log -R
        cd /tmp/$(hostname)_hungsos
        rpm -qa &> rpm-qa
        rpm -Va &> rpm-Va #this command may take a while to run
        cd /tmp
        tar --remove-files -cvjf /tmp/hungsos.$(hostname)_$(date +%Y%m%d%H%M%S).tar.bz2 ./$(hostname)_hungsos
        rm -Rf /tmp/$(hostname)_hungsos 2> /dev/null
        ls -ltr hungsos.$(hostname)_*.tar.bz2 | tail -1
    

OH boy! the following makes me seat thinking about it and I hope to never do it. But then again I do love to learn new stuff.

 

For Azure Virtual Guests Follow the directions in: “How to Create a Microsoft Azure Virtual Machine sosreport and Submit the Report to Red Hat” for Azure virtual guests. Issue How to generate a sosreport? If the file is a large and you are having difficulty uploading it, see the following document for options on how to deal with large files, see following knowledge document: “What to do if the sosreport output file is too large to upload to Support case/Service request through Red Hat Customer Portal?”. Wha

Source: What is a sosreport and how to create one in Red Hat Enterprise Linux 4.6 and later?