Friday, August 29, 2014

OpenStack monitoring: Zabbix Ceilometer proxy installation

Recently a Ceilometer proxy for Zabbix was released by OneSource. This proxy will pull all the instance information from OpenStack and populate it in Zabbix

The source code can be downloaded from here:

The basic prerequisites for the server where the proxy is running is Python and Pika library. Also there should be network connectivity from the proxy machine to your OpenStack installation.

In the test installation, I tried it on a standalone Ubuntu machine. Python can be installed using apt-get and was pretty much straight forward. The document suggests installation of Pika using PIP package manager. Since the machine is sitting behind a proxy, we had some trouble using PIP. As a workaround, the Pika can be directly downloaded and installed from the source repository here

Simply download all files in the repo and execute the script to install pika

Now coming back to the Ceilometer proxy installation. For this you have to uncomment this line in your keystone.conf file.

 notification_driver = keystone.openstack.common.notifier.rpc_notifier
 Next step is to update the proxy.conf file with your OpenStack installation connection parameters.This is also pretty much straight forward for people familiar with OpenStack. I had some confusion on the rabbitmq account to be used, the 'guest' account worked fine for me. Other than that it is just your ceilometer api IP address and keystone authentication details. In the zabbix_configs section, you need to provide your zabbix host IP and admin credentials for web login

Once the proxy.conf file is updated, you can simply run the script to start the monitoring. A new entry for the proxy will be created Administration->DM .

Note: One shortcoming we have noticed is that the instances created using heat orchestrator are not being picked up by the proxy.  Also the cleanup of machines from Zabbix once they are deleted from openstack is not happening.


Friday, August 22, 2014

Agentless openstack monitoring using zabbix

Zabbix can be a tough cookie to crack!! And if you are planning to monitor Openstack using Zabbix, there is lot of additional work to be done .More so, if you want to go the agentless way, ie using SNMP

So, here we go.I am using Ubuntu 12.04 OS, both for my Zabbix server as well as openstack nodes

  • First you need to install the following packages using apt-get in the machine being monitored ie the openstack node

apt-get install snmpd
apt-get install snmp snmp-mibs-downloader

  • snmpd will be installed by default in your zabbix server, but you need to install the snmp and snmp-mibs-downloader packages as well in the server
  •  Once that is done, edit the /etc/snmp/snmpd.conf file in your openstack node. Update the following values
agentAddress udp:161,udp6:[::1]:161
rocommunity public <Ip of your zabbix server>
proc  apache2
proc  neutron-server
proc  nova-api

PS: the process names will depend on the openstack node. Name all the processes that you want to monitor

  • Create the openstack host in the Zabbix server, select snmp interface during the host creation
  • By default zabbix has snmp templates for monitoring disk space , cpu utilization, network interface status and  system uptime. You can attach those template to your host
  • Inorder to monitor memory of system using SNMP, we can make us eof the following OIDs to create new templates
Memory Statistics:
Total Swap Size: .
Available Swap Space: .
Total RAM in machine: .
Total RAM used: .
Total RAM Free: .
Total RAM Shared: .
Total RAM Buffered: .
Total Cached Memory: .

  • For eg, if you want to monitor the total RAM used, first execute the following command from the zabbix server
 snmpwalk -v 2c -c public <openstack node ip> .

You will be getting an output which will look like this]

UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 2420936 kB

In this case memAvailReal.0 is the value you should be using for SNMP OID value in the next step
  • You can clone any of the existing SNMP templates, and create new items . You will have to update the 'key' and 'SNMP OID' value in the new item based on the above output. The Key can be any unique value, make sure that the OIDs match the value mentioned in above step
  • In case you want to monitor a process via snmp, as mentioned earlier, it should be defined on the machine's snmpd.conf. Now execute the following command from the zabbix server 

 snmpwalk -v 2c -c public <openstack node ip> prTable 
  • Output should look something like this

UCD-SNMP-MIB::prNames.1 = STRING: mountd
UCD-SNMP-MIB::prNames.2 = STRING: ntalkd
UCD-SNMP-MIB::prNames.3 = STRING: sendmail
UCD-SNMP-MIB::prNames.4 = STRING: /usr/bin/nova-api
UCD-SNMP-MIB::prNames.5 = STRING: apache2
UCD-SNMP-MIB::prNames.6 = STRING: neutron-server
UCD-SNMP-MIB::prNames.7 = STRING: nova-api

UCD-SNMP-MIB::prErrorFlag.1 = INTEGER: error(1)
UCD-SNMP-MIB::prErrorFlag.2 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.3 = INTEGER: error(1)
UCD-SNMP-MIB::prErrorFlag.4 = INTEGER: error(1)
UCD-SNMP-MIB::prErrorFlag.5 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.6 = INTEGER: noError(0)
UCD-SNMP-MIB::prErrorFlag.7 = INTEGER: noError(0)

Note the prErrorFlag.n field. We will be using this as SNMP OID in the template for process monitoring. The logic to be used, as clear from the output above is that, if the process is up and running the output will be noError(0)


Thursday, August 21, 2014

Tech tip: Increase openstack project quota from command line

1. List the keystone tenants and search for the required tenant

keystone tenant-list |grep <tenantname>

 Note the id of the tenant being displayed. You need to use this id in the next command

2. Get quota details of the tenant using the following command

nova-manage project quota <tenantid>

You will be getting output similar to this

Quota                                Limit      In Use     Reserved
metadata_items                       128        0          0
injected_file_content_bytes          10240      0          0
ram                                  51200      0          0
floating_ips                         10         0          0
security_group_rules                 20         0          0
instances                            10         0          0
key_pairs                            100        0          0
injected_files                       5          0          0
cores                                20         0          0
fixed_ips                            unlimited  0          0
injected_file_path_bytes             255        0          0
security_groups                      10         0          0

3. Update value of the key, depending on which item you want to update. For eg, if you want to increase the number of instances from 10 to 20, give the following command

nova-manage project quota <tenantid> --key instances --value 20

4.Now run the "nova-manage quota <tenantid> " command to see if the quota is updated

Wednesday, August 13, 2014

Instances goes to paused state in Openstack Havanna


All instances in openstack will be in paused node. You will not be able to create new instances or switch on any of the paused instances


Most often the reason will be lack of disk space in your compute node. By default the instances are created in the /var/lib/nova/instances folder of the compute node. This location is defined by the parameter "instances_path" in nova.conf of the  compute node. If your "/" partition is running out of disk space, then you cannot perform any instance related operations


  • Change the "instances_path" location to a different location. Ideally you could attach an additional disk and mount it to a directory and update the directory path in the "instance_path" variable.
  • Problem arises when you already have a number of instances  in the previous folder. You should move them over to the new location.
  •  Also you should set the group and ownership of the new instances folder to "nova" user, so that the permissions, ownership and group memberships are same as that of the previous folder

Openstack havanna neutron agent-list alive status error

In some scenarios, the openstack neutron-agent status will show as xxx even though you could see he neutron agents services are up and running in the network and compute nodes. Also you could see a fluctuation in the agent status if you try the agent-list command repeatedly.  Confusing, right?

Actually  the problem is not in the actual agent status, but with two default configurations in neutron.conf ie agent_down_time and report_interval. It is the interval during which neutron will check the agent status. There is a bug reported against this issue

As per the details in the bug " report_interval" is how often an agent sends out a heartbeat to the service. The Neutron service responds to these 'report_state' RPC messages by updating the agent's heartbeat DB record. The last heartbeat is then compared to the configured agent_down_time to determine if the agent is up or down"

The neutron agent-list command uses the agent_down_time value to display the status. The default values are set very low, because of which the alive status is shown as down/fluctuating.

Solution: As suggested in the solution for the bug, update the values of agent_down_time and report_interval to 75 and 30 seconds respectively. Since the above mentioned rpc issue with open-vswitch agent in compute is  resolved by this, all the agents will be shown as alive

Friday, July 25, 2014

Ubuntu 12.04 P2V conversion using non-root user

Ubuntu P2V conversion is not as straight forward as other Linux machines with a root user. This is because we use a non-root user by default for managing Ubuntu machines and the root credentials are not known to us. So how do you convert a physical Ubuntu VM to virtual without the root credentials? Here are the steps

PS: please note the steps are for VMware vCenter standalone converter 5.5

1.Edit VMware configuration files converter-agent.xml and converter-worker.xml files present in C:\ProgramData\VMware\VMware vCenter Converter Standalone , update the useSudo flag from false to true

2. Reboot the VMware converter standalone agent service

3. On the Physical server that needs to be converted, edit the /etc/sudoers file and add the following entry

<username> ALL=(ALL) NOPASSWD: ALL

4.Ensure that the following entry is not present in /etc/sudoers

Defaults requiretty

5. You need to change the userid and group id of the non-root user to 0. Edit /etc/password and /etc/group file for this

For eg: in /etc/password, update as following for the user


In /etc/group, update as follows


6.In the /etc/ssh/sshd_config . allow root login through ssh

PermitRootLogin yes

7.Now you need to open your standalone convertor as administrator and start the conversion wizard

Networking considerations during the conversion

A helper VM will be created during the conversion process, which will either get an IP from DHCP or you should assign a static IP to you. It will be assigned by default  to the "VM network" port group, though there is option to change it . If your network doesnt have a dhcp, assign a static  IP  to the helper VM and make sure that VMs in the assigned  port group can communicate with the physical server being converted

Monday, July 21, 2014

Tech tip: Create separate routing table for multiple nics in Ubuntu

Scenario: 2 nics in Ubuntu machine, requirement to assign IPs from different VLANs to each of these interfaces, access from outside world to all the assigned IPs.

The situation was a bit complex since the machine was a VM  in ESXi and each of these nics were added to portgroups of two VLANs 200 and 201. The first nic eth0 was assigned a gateway , and was accessible from outside world. The second nic eth1 was assigned IP in 201  VLAN, but was not able to ping to that machine from other machines in a different VLAN


Inorder to solve the issue, we had to add an additional routing table, to select default route for packages which should go out of eth1. The following lines added to the eth1 interface configuration in /etc/networking/interfaces file did the trick

post-up ip route add default via 10.1111.0.1 dev eth1 table 101
post-up ip rule add from lookup 101

Thursday, July 17, 2014

Tech tip: system error 53

Recently we faced a strange issue while connecting a Windows server 2012 R2 to domain, connection was failing with network error.

I was able to ping to the domain, as well as the domain controller. While troubleshooting the issue, I stumbled upon a gem called c:\windows\debug\netsetup.log. It was throwing an error

'NetUseAdd to \\<domain controller>\IPC$ returned 1231

Seems like the machine was having issue connecting to the share. I tried  the "Net view \\sharename" command and got system error 53. Now the root cause was confirmed, the machine was having trouble accessing shared drives

How do we solve it? Actually the solution was pretty simple , you need to install 'client for Microsoft networks" in the network adapter of the server. Somehow it got uninstalled from my VM's adapter and created all this trouble. Installed it back and was able to add the machine to domain!!


Friday, July 4, 2014

Tech tip : How to set http proxy for apt-get in Ubuntu

Steps to set an http proxy for apt-get permanently

1. Create a new file apt.conf in /etc/apt  . Add the following line in it

Acquire::http::proxy "http://<proxyip>:<port>/";

 2. Edit the file named environment in /etc .Add the following line to it.


If you want to add the proxy temporarily for a session, you can use the command

export http_proxy=http://yourproxyaddress:proxyport

PS: if you want to add proxy exception for local addresses in your network, add the following command to /etc/environment

no_proxy = localhost,<hostname>

Friday, April 4, 2014

Create Windows stack using Heat Orchestration Template in Openstack

The blog explains the process of creating a basic Windows stack in Openstack using Heat Orchestration Template. The hypervisor being used is VMware ESXi5.5.

Image preparation to upload in glance

  • Lets start with creating a Windows 2012 R2 VM in ESXi. In this server, download and install the cloudbase-init package for windows .The beta version is available at this link:

Follow the steps in this link for installation :

  • Once installation is completed, edit the 'setup.exe" registry key at HKLocal machine/SYSTEM/SETUP/STATUS/ChildCompletion and change the value from 1 to 3. This is to avoid a system restart exception when the image boots up for the first time in openstack
  • If you want to do any custom configurations in the windows machine, like open a specific firewall port, enable ping ,rdp can do it at this point
  •  Run Syprep and shutdown the VM
 C:\Windows\System32\sysprep\sysprep.exe /generalize /oobe /shutdown

  • Use a VMware standalone convertor ,select the prepared VM as source and convert it to a VM suitable for Vmware workstation 10.0.x. When the conversion process is completed, you will get a vmdk and .vmx file at the destination
  • Using winscp or any other similar tools, copy the converted vmdk to your openstack glance  server. 
  • Create an image from this vmdk using the following command

glance image-create --name <image name>--disk-format=vmdk --container-format=bare --is-public=true --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property vmware_ostype="windows8Server64Guest" < Openstack_win2012.vmdk

Sample yaml template

The template given below uses heat orchestrator to spin up an instance with the image that we created and install IIS in it

heat_template_version: 2013-05-23

description: >
  Basic windows+IIS installation


    type: string
    description : Name of a KeyPair to enable access to the instance
    type: string
    description: Instance type for Windows server
    default: m1.small
      - allowed_values: [m1.small, m1.medium, m1.large]
        description: instance_type must be one of m1.small, m1.medium or m1.large
    type: string
    description: ID of the image to use for the Windows erver
   type: string
   description: windows feature to be installed

    type: OS::Nova::Server
      image: { get_param: image_id }
      flavor: { get_param: instance_type }
      key_name: { get_param: key_name }
          template: |

            Install-WindowsFeature -Name feature -IncludeManagementTools
            feature: { get_param: windows_feature }


 Get password of the instance

Once the instance is up and running, you can use the keypair used while spinning up the stack to retrieve the password. In your openstance machine, run the following command

nova get-password <instance-id> <private key>

instance - id: This is the Id of instance created by the stack , can be obtained from horizon dashboard
privaye key : This is the pem file you downloaded while creating the keypair. Please note that you should copy it over to openstack machine using winscp before running the get-password command


The VMware specific configuration options for creating glance image
The Vmware_ostype derived from the enumerator VirtualMachineGuestOsIdentifier in the API reference: