MAPI - Log Resources

New in version 0.9.3.

This class allows to request internal logs from HCP, which will then prepare a zip-file to be downloaded later on.

Note

To be able to use this class, HCP needs to run at least version 7.2.

The usage pattern is this:

  • prepare()
  • check the status() until readyForStreaming is True, then
  • download() the logs
  • save the zip’ed logs or process them, what ever is needed.
  • close() the underlying Connection() object

Classes

Logs

class hcpsdk.mapi.Logs(target, debuglevel=0)[source]

Access to HCP internal logfiles (ACCESS, SYSTEM, SERVICE, APPLICATION)

Parameters:
  • target – an hcpsdk.Target object
  • debuglevel – 0..9 (used in http.client)
Raises:
hcpsdk.HcpsdkPortError in case target is

initialized with an incorrect port for use by this class.

Class constants:

Log types:

L_ACCESS

The log files http_gateway_request.log.x, mapi_gateway_request.log.x and admin_request.log.x contain all HTTP access messages that includes all of the HTTP based (SMC, TMC, MAPI, MQE, REST, HS3, HSwift) gateway logs. MAPI and Admin will contain all the system level access information while http gateway contains TMC and namespace access info.

L_SYSTEM

This is a snapshot of the running system when the logs were captured. This includes the messages that are generated by HCP OS, the firewall and routing rules, the fiber channel info and many others and are always important as a first step for support to go over.

L_SERVICE

These are logs generated when performing any of the service procedures such as when installing HCP software, adding a node, adding a LUN, node recovery etc.

L_APPLICATION

These are logs from all the HCP specific services such as the JVM, the volume manager, storman etc.

L_ALL

A list containing all other L_* log types.

Class attributes

suggestedfilename

When a download has been started, this attribute holds the filename suggested by HCP.

Class methodes:

mark(message)[source]

Mark HCPs internal log with a message.

Parameters:message – the message to be logged
Raises:LogsError
prepare(startdate=None, enddate=None, snodes=[])[source]

Command HCP to prepare logs from startdate to enddate for later download.

Parameters:
  • startdate – 1st day to collect (as a datetime.date object)
  • enddate – last day to collect (as a datetime.date object)
  • snodes – list of S-series nodes to collect from
Returns:

a tuple of datetime.date(startdate), datetime.date(enddate) and str(prepared XML)

Raises:

ValueError arguments are invalid or one of the LogsError if an operation failed

status()[source]

Query HCP for the status of the request log download.

Returns:a collection.OrderedDict:
{
 readyForStreaming: bool,
 streamingInProgress: bool,
 started: bool,
 error: bool,
 content: list # one or more of: L_ACCESS, L_SYSTEM,
               # L_SERVICE, L_APPLICATION)
}
Raises:re-raises whatever is raised below
download(hdl=None, nodes=[], snodes=[], logs=[], progresshook=None, hidden=True)[source]

Download the requested logs.

Parameters:
  • hdl – a file (or file-like) handle open for binary read/write or None, in which case a temporary file will be created
  • nodes – list of node-IDs (int), all if empty
  • snodes – list of S-node names (str), none if empty
  • logs – list of logs (L_*), all if empty
  • progresshook – a function taking a single argument (the # of bytes received) that will be called after each chunk of bytes downloaded
  • hidden – the temporary file created will be hidden if possible (see tempfile.TemporaryFile())
Returns:

a 2-tuple: the file handle holding the received logs, positioned at byte 0 and the filename suggested by HCP

Raises:

LogsError or LogsNotReadyError

cancel()[source]

Cancel a log request.

Returns:True if cancel was successfull
Raises:LogsError in case the cancel failed
close()[source]

Close the underlying hcpsdk.Connection().

Exceptions

exception hcpsdk.mapi.LogsError(reason)[source]

Base Exception used by the hcpsdk.mapi.Logs() class.

Parameters:reason – An error description
exception hcpsdk.mapi.LogsNotReadyError(reason)[source]

Raised by Logs.download() in case there are no logs ready to be downloaded.

Parameters:reason – An error description
exception hcpsdk.mapi.LogsInProgessError(reason)[source]

Raised by Logs.download() in case the log download is already in progress.

Parameters:reason – An error description

Sample Code

The following example code creates a simple command processor that allows to prepare and download logs from HCP.

# -*- coding: utf-8 -*-
# The MIT License (MIT)
#
# Copyright (c) 2014-2018 Thorsten Simons (sw@snomis.de)
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

import sys
from pprint import pprint
import logging
import cmd
from datetime import date, timedelta
import hcpsdk

USR = 'logmon'
PWD = 'logmon01'
TGT = 'admin.hcp72.archivas.com'
PORT = hcpsdk.P_MAPI


class LogsShell(cmd.Cmd):
    intro = 'HCP Log Download Shell.   Type help or ? to list commands.\n'
    prompt = '==> '

    def preloop(self):
        self.nodes = []
        self.snodes = []
        self.logs = []
        self.start = date.today() - timedelta(days=7)
        self.end = date.today()

    def do_what(self, arg):
        'what - show what is selected for download'
        print('start date:      {}'.format(self.start.strftime('%Y/%m/%d')))
        print('end date:        {}'.format(self.end.strftime('%Y/%m/%d')))

        print('selected nodes:  {}'.format(' '.join(self.nodes) or \
                                           ' '.join([x.split('.')[3]
                                                     for x in l.target.addresses])))
        print('selected snodes: {}'.format(' '.join(self.snodes) or 'none'))
        print('selected logs:   {}'.format(' '.join(self.logs) or \
                                           ' '.join(hcpsdk.mapi.Logs.L_ALL)))

    def do_status(self, arg):
        'status - query HCP for the log preparation status'
        log.debug('do_status() called')
        try:
            pprint(l.status())
        except Exception as e:
            print(e)

    def do_prepare(self, arg):
        'prepare - trigger HCP to prepare the logs for later download'
        try:
            l.prepare(snodes=self.snodes, startdate=self.start,
                      enddate=self.end)
        except Exception as e:
            print('prepare failed: {}'.format(e))
        else:
            print('preparing for nodes: {}\n'
                  '             snodes: {}\n'
                  '         date range: {} - {}'
                  .format(' '.join(self.nodes) or 'all',
                          ' '.join(self.snodes) or 'none',
                          self.start.strftime('%Y/%m/%d'),
                          self.end.strftime('%Y/%m/%d')))

    def do_nodes(self, arg):
        'nodes [node_id,]* - select the nodes to download logs from\n'\
        '                    (no argument selects all nodes)'
        if arg:
            self.nodes = []
            n = [x.split('.')[3] for x in l.target.addresses]
            for x in arg.split(','):
                if x in n:
                    self.nodes.append(x)
                else:
                    print('invalid: {}'.format(x))
        else:
            self.nodes = []

    def do_snodes(self, arg):
        'snodes [snode_name,]* - select the S-nodes to download logs from\n'\
        '                        (no argument disables S-Node log download)'
        if arg:
            self.snodes = arg.split(',')
        else:
            self.snodes = []

    def do_logs(self, arg):
        'logs [ACCESS,|SYSTEM,|SERVICE,|APPLICATION]* - select log types\n'\
        '                                               nothing selects all'
        if arg:
            self.logs = []
            for x in [y.upper() for y in arg.split(',')]:
                if x in hcpsdk.mapi.Logs.L_ALL:
                    self.logs.append(x)
                else:
                    print('invalid: {}'.format(x))
        else:
            self.logs = []

    def do_start(self, arg):
        'start YYYY/MM/DD - select start date (default is a week ago)'
        try:
            d = arg.split('/')
            self.start = date(int(d[0]),int(d[1]),int(d[2]))
        except Exception as e:
            print('invalid input - YYYY/MM/DD required...')

    def do_end(self, arg):
        'end YYYY/MM/DD - select end date (default is today)'
        try:
            d = arg.split('/')
            self.end = date(int(d[0]),int(d[1]),int(d[2]))
        except Exception as e:
            print('invalid input - YYYY/MM/DD required...')


    def do_download(self, filename, ):
        'download <filename> - donwload logs and store into file <filename>'
        if not filename:
            print('Error: filename missing!')
            return

        print('downloading for nodes: {}\n'
              '               snodes: {}\n'
              '                 logs: {}\n'
              '           date range: {} - {}\n'
              .format(' '.join(self.nodes) or \
                      ' '.join([x.split('.')[3] for x in l.target.addresses]),
                      ' '.join(self.snodes) or 'none',
                      ' '.join(self.logs) or \
                      ' '.join(hcpsdk.mapi.Logs.L_ALL,),
                      self.start.strftime('%Y/%m/%d'),
                      self.end.strftime('%Y/%m/%d')))
        try:
            with open(filename, 'w+b') as outhdl:
                hdl, n = l.download(hdl=outhdl, nodes=self.nodes,
                                    snodes=self.snodes, logs=self.logs,
                                    progresshook=showdownloadprogress)
                print('\nHCP suggested filename: {}'.format(n))
        except Exception as e:
            print('Error: {}'.format(e))
        else:
            print('download finished')

    def do_cancel(self, arg):
        'cancel - abort a log preparation'
        try:
            l.cancel()
        except Exception as e:
            print('cancel failed: {}'.format(e))
        else:
            print('cancel done')

    def do_mark(self, arg):
        'mark - mark HCPs log with a message'
        try:
            l.mark(arg)
        except Exception as e:
            print('mark failed: {}'.format(e))
        else:
            print('log marked')

    def do_quit(self, arg):
        'quit - exit the HCP Logs Shell'
        self.close()
        print('Bye...')
        return True

    def do_debug(self, arg):
        'debug - Toggle debug output'
        if log.getEffectiveLevel() != logging.DEBUG:
            log.setLevel(logging.DEBUG)
            log.debug('debug enabled')
            print('debug enabled')
        else:
            log.setLevel(logging.CRITICAL)
            print('debug disabled')

    def emptyline(self):
        'run the status command if no command is given'
        return self.onecmd('status')

    def close(self):
        'close the underlying *hcpsdk.Logs()* object'
        l.close()


def showdownloadprogress(nBytes):
    '''
    Print a simple progress meter
    '''
    sz = ["B", "kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"]
    i = 0
    while nBytes > 1023:
                    nBytes = nBytes / 1024
                    i = i + 1
    print("\rreceived: {0:.2f} {1:}".format(nBytes, sz[i]), end='')


if __name__ == '__main__':

    auth = hcpsdk.NativeAuthorization(USR,PWD)
    try:
        t = hcpsdk.Target(TGT, auth, port=PORT)
    except hcpsdk.ips.IpsError as e:
        sys.exit('Can\'t resolve "{}"\n\t==> {}'.format(TGT, e))
    l = hcpsdk.mapi.Logs(t, debuglevel=0)

    # create console handler with a higher log level
    sh = logging.StreamHandler(sys.stderr)
    fh = logging.Formatter("[%(levelname)-8s]: %(message)s")
    sh.setFormatter(fh)
    log = logging.getLogger()
    log.addHandler(sh)
    # this makes the logger silent, until *debug* has activated
    log.setLevel(logging.CRITICAL)


    LogsShell().cmdloop()

Sample Output

HCP Log Download Shell.   Type help or ? to list commands.

==> logs access
==> snodes snode70
==> start 2015/09/23
==> what
start date:      2015/09/23
end date:        2015/09/25
selected nodes:  176
selected snodes: snode70
selected logs:   ACCESS
==> prepare
preparing for nodes: all
             snodes: snode70
         date range: 2015/09/23 - 2015/09/25
==> status
{'readyForStreaming': False,
 'streamingInProgress': False,
 'error': False,
 'started': True,
 'content': ['ACCESS', 'SYSTEM', 'SERVICE', 'APPLICATION']}
==> status
{'readyForStreaming': True,
 'streamingInProgress': False,
 'error': False,
 'started': True,
 'content': ['ACCESS', 'SYSTEM', 'SERVICE', 'APPLICATION']}
==> download logs.zip
downloading for nodes: 176
               snodes: snode70
                 logs: ACCESS
           date range: 2015/09/23 - 2015/09/25

received: 11.41 MB
download finished
==> cancel
cancel done
==> quit
Bye...