Skip to main content


Showing posts from 2014

OpenStack icehouse installation error : nova-api service getting stopped

While trying to install OpenStack icehouse, faced an issue with nova-api service.It was not getting started. The following error was coming up in the Nova-api log Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c Exit code: 1 .......  nova Stdout: '' 2014-10-17 07:21:08.058 27270 TRACE nova Stderr: 'Traceback (most recent call last):\n  File "/usr/bin/nova-rootwrap", line 6, in <module>\n    from oslo.rootwrap.cmd import main\nImportError: No module named rootwrap.cmd\n' Problem was with one of the oslo.rootwrap module. It was broken Solution is to upgrade the module using pip   #pip install oslo.rootwrap --upgrade

OpenStack: Restrict instance deletion

In OpenStack, by default users who are members of a  tenant can delete all instances in that tenant, even if it is spinned up by other users. If you want to restrict that, you  need to tweak the nova policy file  ie /etc/nova/policy.json Add the following lines in the file:     "admin_or_user":"is_admin:True or user_id:%(user_id)s",     "compute:delete":"rule:admin_or_user", Make the same changes in the /etc/openstack-dashboard/nova_policy.json file also Now restart the openstack-nova-api service Now user will be able to delete only those instances spinned up by them. Admin users will be able to delete all instances

OpenStack : Assign floating IP using heat template

Creating Yaml templates that assign floating IPs to your instances being spawned can be a bit tricky.Let us look at a scenario where we need to spin up a VM, assign a floating IP from a pool and make reference to this floating IP in your userdata as well. We will make use of the network ID of the internal and external network, as well as the Subnet ID of the internal network The logical workflow is as follows:   Create a port resource using internal network and internal subnet IDs Create a floating IP resource , referring to the external network ID   Associate the floating IP to the port   In the server resource being created, associate the port resource   Now we will see how this can be implemented using both HOT and AWS template formats

OpenStack monitoring: Zabbix Ceilometer proxy installation

Recently a Ceilometer proxy for Zabbix was released by OneSource. This proxy will pull all the instance information from OpenStack and populate it in Zabbix The source code can be downloaded from here: The basic prerequisites for the server where the proxy is running is Python and Pika library. Also there should be network connectivity from the proxy machine to your OpenStack installation.

Agentless openstack monitoring using zabbix

Zabbix can be a tough cookie to crack!! And if you are planning to monitor Openstack using Zabbix, there is lot of additional work to be done .More so, if you want to go the agentless way, ie using SNMP So, here we go.I am using Ubuntu 12.04 OS, both for my Zabbix server as well as openstack nodes First you need to install the following packages using apt-get in the machine being monitored ie the openstack node

Instances goes to paused state in Openstack Havanna

Issue:  All instances in openstack will be in paused node. You will not be able to create new instances or switch on any of the paused instances Reason:  Most often the reason will be lack of disk space in your compute node. By default the instances are created in the /var/lib/nova/instances folder of the compute node. This location is defined by the parameter "instances_path" in nova.conf of the  compute node. If your "/" partition is running out of disk space, then you cannot perform any instance related operations Solution:  Change the "instances_path" location to a different location. Ideally you could attach an additional disk and mount it to a directory and update the directory path in the "instance_path" variable.   Problem arises when you already have a number of instances  in the previous folder. You should move them over to the new location.  Also you should set the group and ownership of the new instances folder to &quo

Openstack havanna neutron agent-list alive status error

In some scenarios, the openstack neutron-agent status will show as xxx even though you could see he neutron agents services are up and running in the network and compute nodes. Also you could see a fluctuation in the agent status if you try the agent-list command repeatedly.  Confusing, right? Actually  the problem is not in the actual agent status, but with two default configurations in neutron.conf ie agent_down_time and report_interval . It is the interval during which neutron will check the agent status. There is a bug reported against this issue As per the details in the bug " report_interval" is how often an agent sends out a heartbeat to the service. The Neutron service responds to these 'report_state' RPC messages by updating the agent's heartbeat DB record. The last heartbeat is then compared to the configured agent_down_time to determine if the agent is up or down" The neutron agent-list comm

Ubuntu 12.04 P2V conversion using non-root user

Ubuntu P2V conversion is not as straight forward as other Linux machines with a root user. This is because we use a non-root user by default for managing Ubuntu machines and the root credentials are not known to us. So how do you convert a physical Ubuntu VM to virtual without the root credentials? Here are the steps PS: please note the steps are for VMware vCenter standalone converter 5.5 1.Edit VMware configuration files converter-agent.xml and converter-worker.xml files present in C:\ProgramData\VMware\VMware vCenter Converter Standalone , update the useSudo flag from false to true 2. Reboot the VMware converter standalone agent service

Tech tip: Create separate routing table for multiple nics in Ubuntu

Scenario: 2 nics in Ubuntu machine, requirement to assign IPs from different VLANs to each of these interfaces, access from outside world to all the assigned IPs. The situation was a bit complex since the machine was a VM  in ESXi and each of these nics were added to portgroups of two VLANs 200 and 201. The first nic eth0 was assigned a gateway , and was accessible from outside world. The second nic eth1 was assigned IP in 201  VLAN, but was not able to ping to that machine from other machines in a different VLAN Solution: Inorder to solve the issue, we had to add an additional routing table, to select default route for packages which should go out of eth1. The following lines added to the eth1 interface configuration in /etc/networking/interfaces file did the trick post-up ip route add default via 10.1111.0.1 dev eth1 table 101 post-up ip rule add from lookup 101

Tech tip: system error 53

Recently we faced a strange issue while connecting a Windows server 2012 R2 to domain, connection was failing with network error. I was able to ping to the domain, as well as the domain controller. While troubleshooting the issue, I stumbled upon a gem called c:\windows\debug\netsetup.log. It was throwing an error 'NetUseAdd to \\<domain controller>\IPC$ returned 1231 Seems like the machine was having issue connecting to the share. I tried  the "Net view \\sharename " command and got system error 53. Now the root cause was confirmed, the machine was having trouble accessing shared drives How do we solve it? Actually the solution was pretty simple , you need to install 'client for Microsoft networks" in the network adapter of the server. Somehow it got uninstalled from my VM's adapter and created all this trouble. Installed it back and was able to add the machine to domain!!  

Tech tip : How to set http proxy for apt-get in Ubuntu

Steps to set an http proxy for apt-get permanently 1. Create a new file apt.conf in /etc/apt  . Add the following line in it Acquire::http::proxy " http://<proxyip>:<port>/" ;  2. Edit the file named environment in /etc .Add the following line to it. http_proxy= http://<proxyip>:<port> If you want to add the proxy temporarily for a session, you can use the command export http_proxy=http://yourproxyaddress:proxyport PS: if you want to add proxy exception for local addresses in your network, add the following command to /etc/environment no_proxy = localhost,<hostname>

Create Windows stack using Heat Orchestration Template in Openstack

The blog explains the process of creating a basic Windows stack in Openstack using Heat Orchestration Template. The hypervisor being used is VMware ESXi5.5. Image preparation to upload in glance Lets start with creating a Windows 2012 R2 VM in ESXi. In this server, download and install the cloudbase-init package for windows .The beta version is available at this link: Follow the steps in this link for installation : Once installation is completed, edit the 'setup.exe" registry key at HKLocal machine/SYSTEM/SETUP/STATUS/ChildCompletion and change the value from 1 to 3. This is to avoid a system restart exception when the image boots up for the first time in openstack If you want to do any custom configurations in the windows machine, like open a specific firewall port, enable ping ,rdp can do it at this point  Run Syprep and shu

How to create new Local SR in XenServer

In this blog, I will explain the process of creating an additional local SR in XenServer. In this scenario, there is an additional hard disk present in the server which I plan to add as a local SR in addition to the existing Local storage First you need to identify which disk is being currently used for your existing local SR, so that it is not accidentally deleted 1)SSH to your server using putty type the command xsconsole In the console that is displayed go to Disks & Storage repositories -> Current Storage Repositories You need to locate your current local repository and view the properties on the right pane Note down the scsi ID 2)Type the following command #ll /dev/disk/by-id Note down which device the ID maps to. In this case it is sda3.So that is your present local SR Also you need to note the SCSI id of the disk that you are planning to use for the new SR. In my case it is sdb 3)Next step is to create the new SR. You need run

Install nested KVM in VMware ESXi 5.1

In this blog, I will explain the steps required to run a nested KVM hypervisor on  Vmware ESXi. The installation of KVM is done on Ubuntu 13.10(64 bit). Note: It is assumed that you have already installed your Ubuntu 13.10 VM in ESXi, and hence we will not look into the Ubuntu installation part. 1) Upgrade VM Hardware version to 9. In my ESXi server, the default VM hardware version was 8. So I had to shutdown my VM and upgrade the Hardware version to 9 to get the KVM hypervisor working. You can right click the VM and select the Upgrade hardware option to do this. 2)In the ESXi host In /etc/vmware edit the 'config' file and add the following setting vhv.enable = "TRUE" 3)Edit the VM settings and go to VM settings > Options  > CPU/MMU Virtualization . Select the Intel EPT option 4) Go to Options->CPUID mask> Advanced-> Level 1, add the following CPU mask level ECX  ---- ---- ---- ---- ---- ---- --H- ---- 5) Open the vmx

Timeout error during OpenStack installation

While trying to install OpenStack using RDO, following the instructions in the link here , you might face the following timeout error Applying <IP address>_prescript.pp   [ ERROR ] ERROR : Error appeared during Puppet run: <IP address>_prescript.pp Error: Command exceeded timeout The reason for this error, in all probabilities would be the delay in your network, which could be causing the netns component packages to be timed out You can increase the timeout by editing the following file /usr/lib/python2.6/site-packages/packstack/puppet/modules/packstack/manifests/netns.pp   Add a timeout line in the file     if $::netns_support != "true" {         exec { "netns_dependecy_install":             path => "/usr/bin/",             command => "yum update -y kernel iputils iproute",             timeout => 1900 ,         } Now you can retry the installation. It is advisable to use the answer file that is cre

XenCenter troubleshooting

Issue 1: While trying to add a new host to a Pool in XenCenter, I was getting error "internal connection failed no route to host " Solution: It was quite simple actually, but since I couldnt find it mentioned in any knowledge bases or articles, I am noting it down here. The machine where XenCenter was installed couldnt resolve the Xen host name. The machines were not added to domains and hence no DNS entries were there. So I had to add a host entry to the XenCenter machine to help with the name resolution Issue 2: Enabling AD authentication via XenCenter fails with error "clock skew detected with active directory server" Reason: This happens because of the time difference between XenServer and AD server Solution: Update NTP server list of the XenServer and point it to the AD server of the domain. Steps below From the XenServer console, go to 'Network and management interface"->Network time(NTP) Select option "Remove all NTP se