Using VMware vCenter Log Insight with Symmetrix VMAX

A bit of a long one here. At some point this might turn into a white paper (update: it is now). But for now…

Check out my post on the Pure Storage integration with Log Insight here!

*************************************************

UPDATE: We have released a content pack that automatically configures dashboards and fields for the VMAX, it will save you a lot of work and the pack is free! Read about it here:

https://www.codyhosterman.com/2013/08/09/the-vmax-content-pack-for-vmware-vcenter-log-insight/

And updated here:

http://drewtonnesen.wordpress.com/2014/02/12/auditing-with-the-vmax-content-pack-v1-5-1/

*************************************************

Earlier this summer VMware announced a new product called vCenter Log Insight which just went GA today. You can download it and try it out from here:

http://www.vmware.com/products/datacenter-virtualization/vcenter-log-insight/

Log Insight is pretty much exactly what the name indicates, a log aggregator that allows for much easier perusal, consumption and analysis of the myriad of logs in a VMware environment. This can be a very useful tool when trying to perform a root cause analysis of an event where you may not know what product caused the failure. To compound the usefulness you can automatically send your Symmetrix events into it!

Solutions Enabler has a built-in capability to monitor the Symmetrix event log and send all of those event messages to a remote syslog server (or a file, or SNMP etc.). Besides monitoring VMware product logs, VMware vCenter Log Insight has the ability to act as a remote syslog server as well. Therefore, Symmetrix events can be viewed and analyzed from the comfort of Log Insight too!

Solutions Enabler is an API/CLI tools that consists of multiple daemons that each serve a specific purpose. The “storsrvd” daemon is a CLI server for receiving remote CLI commands, “storrdfd” is a daemon that controls RDF consistency etc. There are many more, but the one that is relevant to this topic is the daemon named “storevntd”–the Event Daemon.

By default, Solutions Enabler (and storevntd) does not issue events to a remote syslog server. This has to be configured first. Storevntd is a very powerful tool and can be customized in many ways to send as many or as little events as you want. I will go through some basic configuration here, but for more options and information check out the Solutions Enabler Installation Guide. It has all of the information you need for advanced configuration of the storevntd daemon.

Once you have Solutions Enabler installed (and gatekeepers presented) you can configure syslogging. First install the storevntd if not already done–I recommend enabling autostart as well so it will start back up automatically when/if the server is rebooted. Issue the following command:

stordaemon install storevntd -autostart

The behavior of the storevntd (like all daemons) is controlled by a file named “daemon_options”. The location of this file changes according to operating system so check the SE Install Guide for the location of yours. I am using Windows so it is at C:Program FilesEMCSYMAPIconfig.

There are a ton of options for storevntd but I will focus on the basic ones needed to get syslog up and running.

These are:

storevntd:LOG_EVENT_TARGETS 

storevntd:LOG_EVENT_SYSLOG_HOST

storevntd:LOG_EVENT_SYSLOG_PORT

storevntd:LOG_SYMMETRIX_EVENTS

The event targets option indicates to storevntd which type of message should it be issuing. For syslog, simply configure it to syslog (make sure to uncomment it out by removing the #).

storevntd:LOG_EVENT_TARGETS = syslog

Host and port are simply the syslog host name or IP address of the Log Insight server and the port it listens at. VMware vCenter Log Insight listens at ports 514 TCP, 1514 TCP and 514 UDP. My settings looks like this:

storevntd:LOG_EVENT_SYSLOG_HOST = 192.168.160.153
storevntd:LOG_EVENT_SYSLOG_PORT = 514

One thing I want to note is that make sure to not have white space after the IP address or host name of your target syslog host. I accidentally had a space there once and it wouldn’t forward syslog messages and I got the following somewhat confusing errors in the storeventd log:

<Error> [1832 PDS-DPC-0] Mar-19 15:52:15.547 pdsSockGetAddrs #3615 : Lookup of 192.168.160.153 storevntd:log_event_syslog_port = 514 [via getaddrinfo()] [] failed, h_errno/error_num= 11001
<Error> [1832 PDS-DPC-0] Mar-19 15:52:15.547 : [evtd_logOptionsReload] Unable to lookup Syslog host (option log_event_syslog_host = 192.168.160.153

If you see something like that verify the IP/host is immediately followed by a carriage return, not a space.

The last option is somewhat more involved. Symmetrix Events dictates what events are forwarded (what array, what components, what devices etc.). My option looks like so:

storevntd:LOG_SYMMETRIX_EVENTS = sid=000195701248, status, groups, optimizer, events, array subsystem, checksum, diagnostic, environmental, device pool, service processor, srdf system, srdf link, srdfa session, srdf consistency group, director, device, disk, audit ;
sid=000195701238, status, groups, optimizer, events, array subsystem, checksum, diagnostic, environmental, device pool, service processor, srdf system, srdf link, srdfa session, srdf consistency group, director, device, disk, audit ;
sid=000198700582, status, groups, optimizer, events, array subsystem, checksum, diagnostic, environmental, device pool, service processor, srdf system, srdf link, srdfa session, srdf consistency group, director, device, disk, audit ;

NOTE: If you want to also use the Unisphere syslog feature on top of this you must add the category smc (for general Unisphere alerts) and spa (for performance alerts). Unisphere functionality is a post for another time though.

As you can see there are three arrays listed (you must use the full 12 digits) with a bunch of categories after each. I have included all of the categories that storevntd offers for each. You can choose to not include an array ID at all and then all VMAX arrays (local and remote) will be reported. By including array IDs only those listed will be forwarded. Furthermore you can filter what is sent, by severity, devices, pools etc. I would recommend trying to limit what is sent (especially in large environments) to what is in use by the VMware environment.

In my case I am forwarding all event categories for array 1238, 1248 and 0582.

Once you have these configured the way you want, save the options file and restart the daemon. At this point, Solutions Enabler will begin to forward along syslog messages to Log Insight! A screenshot of Symmetrix messages in Log Insight is below:

From here you can now begin to customize Log Insight to be a bit more friendly with Symmetrix syslog messages. Most messages contain one or more of the following pieces of information:

• Format–which is usually an event (evt)
• Evtid–this is the event type ID. To find a list of all event IDs run the command:

                 stordaemon action storevntd -cmd list -events

• Date–Date/time of event
• Symid–Symmetrix serial number
• Device–Symmetrix device ID
• Sev–Severity of event

VMware vCenter Log Insight allows you to search and sort by default fields and ones you create yourself. A simple one that is pre-created is called hostname. You can choose the Solutions Enabler hostname (or IP address) and choose to see just those messages from that host. But more importantly you can drill down even further by creating your own.

Let’s say you want to see all of the events from just a certain array. To achieve this you need to extract a new object and create a new designated field. Find a message from Solutions Enabler in Log Insight from the Interactive Analytics page and highlight the full array ID as shown below:

You will notice that below the highlight a green message pops up that says “Extract Field”. Click it. On the right a new pane appears that allows you to create this new field which allows Log Insight to recognize array IDs.

Play with the drop down in the value to get what you want. Since the array ID is always an integer I chose that. The -?/d+ is the “code” that means integer. Using the Log Insight User Guide you can create far more complex inputs though. But for array ID this will suffice. The next part is important–the context. The context inputs allow you to say what Log Insight should expect to see before and after this field. This allows it to differentiate between integers from other applications that might be similar to an array ID. In these messages the array ID is always preceded by symid= and followed by ]. This tells Log Insight that any integer between these strings should be considered a value belonging to this field. Lastly you can give it a name–I chose VMAX SN. Go ahead and click save/update.

Now below this a new field will appear along with the original ones like hostname and source. VMAX SN. Expand this field and it will show you a neat little chart of the messages from each one of your arrays.

You can now sort by array serial number! Using this same methodology you can do this with anything else that appears in the syslog messages. Be creative!

If you click on one of the bars the top pane with the graph will change to that field. Below is an image from a new field I created that sorts by Symmetrix Event ID:

Now go to the “Dashboards” page and you can create a new custom dashboard. I named mine “Symmetrix Info”. Once the dashboard is created you can go back to Interactive Analysis and select your custom fields to they appear above. In the above graph look in the top left for “Add to Dashboard”. Click that and add it to you new dashboard. I am adding my Event ID field and the Symmetrix Serial Number field.

If you navigate back to your dashboard you will see your added event fields:

I just started using this and I think it is pretty cool. I am sure there is a lot more to do with it and I have just barely scratched the surface when it comes to playing with creating new fields. Hopefully more posts on this to come!