Symmetrix VMAX and Splunk introduction

My colleague Jonas Rosland (@virtualswede) has been doing some great work with Splunk recently (check out a post from his blog here) and it got me to want to get my own hands a little dirty. So far I’ve only scratched the surface on what can be done with Splunk but I decided to put a post together on some basics. For this post I am going to talk about Symmetrix VMAX logs and how to get those into Splunk.

First off, what is Splunk? If you are familiar with VMware Log Insight it is a very similar idea; a log aggregation tool for analysis, reporting and management. That is definitely a far too simplistic view though in my opinion. So I will take Splunk’s own description right from their website (www.splunk.com):

“Splunk was founded to pursue a disruptive new vision: make machine data accessible, usable and valuable to everyone. Machine data is one of the fastest growing and most pervasive segments of “big data”—generated by websites, applications, servers, networks, mobile devices and all the sensors and RFID assets that produce data every second of every day. By monitoring and analyzing everything from customer clickstreams and transactions to network activity and call records—and more—Splunk turns machine data into valuable insights no matter what business you’re in. It’s what we call Operational Intelligence.”

If you are familiar with the Symmetrix and its surrounding ecosystem you know that there are tons of logs from tons of places. Whether they be Solutions Enabler logs, Unisphere logs, VSI logs, SRDF SRA etc. etc. etc. In order for Splunk to be able to do what it does best you need to first find a way to get these various logs into the Splunk software to be archived, managed and analyzed. Splunk offers a variety of ways to do this.

First off, the most common approach (which I have heavily used in my Log Insight environment) is the ability for Solutions Enabler to act as a syslog source. For details on setting this up, check out this previous blog post here. That post explains how to set up the SE daemon, storevntd (Event Daemon) and how to send out syslog information to a syslog target. In short I added this configuration information to my SE daemon_options file to send out all kinds of information about my Symmetrix VMAX with serial number 1238 to my Splunk master 10.10.82.63:

storevntd:LOG_EVENT_SYSLOG_HOST = 10.10.82.63
storevntd:LOG_EVENT_TARGETS = syslog
storevntd:LOG_SYMMETRIX_EVENTS = sid=000195701238, status, groups, optimizer, events, array subsystem, checksum, diagnostic, environmental, device pool, service processor, srdf system, srdf link, srdfa session, srdf consistency group, director, device, disk, audit ;

Change your host and serial number, restart the storevntd daemon and you are halfway there.

By default, Splunk does not accept syslog messages from anywhere, so you have to enable this functionality. There are a variety of ways to do this, but the simplest is through the Splunk web interface. First log into the Splunk web interface and on the home screen click the Add Data button on the right:

Then choose “From a UDP port”. Solutions Enabler sends syslog messages by default over UDP to port 514. The port is configurable, but whether it is TCP or UDP on the SE side does not seem to be a configurable option. I will look into that to see if TCP is possible, but for now choose UDP within Splunk.

The next screen allows you to configure Splunk to open up a certain port to accept syslog messages on. I am going to use the default SE port of 514 and change the drop down for source type to syslog. Other than that you do not have to change anything else. But you certainly can–for instance the default is to accept syslog from any host, but you can restrict it to a given host. Or you can have Splunk give the host a special name etc. For this I am only changing the port and the source type. Keep it simple.

Save the config and you are good to go! SE should start sending messages to Splunk. You can search a myriad of ways and perfecting the possibilities of the search syntax is a masters course in and of itself so for the purposes of this post I will show an example by selecting the source and showing the recent messages. This can be seen in the “Search & Reporting” page by clicking on the Data Summary box and selecting the port 514 UDP source.

Once clicked it will load all of the syslog messages from that source onto the screen. The great thing is that Splunk is smart enough to recognize fields and automatically label and allow you to sort by them–you can of course make your own. Fields such as Symmetrix SN (symid) or device ID (device).

Cool stuff. What about everything else? All of those logs that cannot be syslogged without some 3rd party application? Splunk can help you there too. So I am going to pick the SYMAPI log for this example. While the storevntd does a good job of pushing a lot of events, it doesn’t do justice to what could be going on with SYMAPI and SYMCLI. There is a lot of detail that is left behind by the syslog feature.

In this case, I have Solutions Enabler installed on a Windows Server 2008 R2 host and I want my SYMAPI logs to be imported into Splunk. So I installed Splunk on that Windows server and configured it to be a peer to my original Splunk instance. To do this you need to enable clustering and peer forwarding and receiving on the Splunk systems. I configured this new Splunk instance to be a peer.

To get it to watch and import my SYMAPI logs I logged into the new Splunk instance web interface and added the log like so:

1. Go to the “Data Inputs” and this time choose “Files & directories”

2. Select “New” and create a new log operation. You can either choose a specific log or a directory. Indicate the file or the directory in the wizard and the save the new configuration. In this example I will just choose today’s SYMAPI log, but since that log name changes every day you might want to choose the directory.

Finish the wizard and old and new events will begin to appear in Splunk. Below is a screen shot of SYMAPI events in Splunk that recorded operations taken within SYMAPI from the SMI-S Provider when I provisioned a new LUN using VSI USM. The portion of it shown reports the operation of mapping the new device to four directors.

The same can easily be done with something like the SRDF SRA log too. Install Splunk on the SRM server and point it to the C:ProgramDataVMwareVMware vCenter Site Recovery ManagerLogsSRAsEMC Symmetrix directory:

 

Pretty straight forward! Definitely looking forward to diving into this a lot more.