Grid control therapy

I've had a lot of bad experiencing installing and managing the OEM grid control.  Today has been the worst day ever and - as an alternative to self-mutilation - I decided that I would document my woes as they occur as a sort of therapy

 

Here's the background to my latest woes:

On Monday I realized that the database that holds my test grid control repository was severly corrupt: a bunch of ora-600s with internal codes that were associated with rollback segment corruption.  Since the DB had no "real" data and I was lacking a recent backup I decided to rebuild the (RAC cluster) database.

The rebuild was no drama, but since this DB had my grid control repository I decided I'd better re-install grid control as well.  That's when everything went south.

The installation would proceed normally without errors until it reported errors configuring the agent. 

Logs showed that the agent was installed apparently OK,but that the management server was no-where to be found.  opmn status showed every other friggin process but not even an entry for the OMS.  The SYSMAN schema had not even been properly installed.  Somehow, installing the OMS for the second time was screwing things up.

[[email protected] ~]$ emctl status oms
Oracle Enterprise Manager 10g Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.

There's no trace files indicating OMS has ever really started (No emoms.trc for instance).

The only logs are emctl logs - created when I do a status request - with AgentStatus.pm: Unknown command encountered.

So next I tried tearing everything down and installing grid control with a new database, thinking that surely this would avoid the apparent "I don't need to install a DB" problem.  No luck.  No database created - no attempt to start OMS.

So I tried renaming the /etc/oraInst.loc file so my installer was unaware of all previous installs. Sigh.  Same result.   

I spent many hours scouring technet, metalink and googling.... eventually found http://forums.oracle.com/forums/thread.jspa?threadID=337544&start=15&tstart=0 in which a bunch of poor souls who have encountered this problem commiserate.  Suggestions from there include:

 
• Remove any 'legacy' listeners (9i)
 
• This mysterious sequence:

echo `find / -name libdb.so.2 -print` >> /etc/ld.so.conf
find . -name libdb.so.2 -print
.......
vi /etc/ldso.conf
- add result from find
ldconfig -v  (run as root)                   

 
• clean up /etc/hosts (remove ipv6 and check hostname is exactly right)
 
• turn off seLinux
 
• remove all symlinks in the installation directory path
   

The last one really rang a bell with me, since only a few weeks ago I had in fact symlinked the oracle home to a new filesystem.  So....

Hooray!  Avoiding the symlinks allowed the OMS configuration to proceed.  Still had many hours of trying to install:  first had to remove SYSMAN and MGMT_VIEW schemas (see metalink note 358627.1).  Then I had mysterious communication failures during the final phase of opmn setup.  Sigh.  Being as it was close to midnight I blew it all away in the hope that a straight forward install would now work and (at about 1am) .... Kapla!  (Klingon for Victory: see http://www.khemorex-klinzhai.de/e/Hol/). 

Alls well that ends well?

This isn't the first time I've struggled with grid control install. My original attempt to install on a windows system never succeeded and while my previous linux based control worked fine most of the time, I still must have spend many hours fiddling with configuration files to make everything talk together properly.

I'm hardly an unbiased source - our Spotlight and Foglight products are sometimes seen to be competitive so you might want to take my opinions with a grain of salt.  However, I've become convinced that while Grid control and OEM undoubtedly reduce DBA overheads (though I think in some areas - particularly diagnostics and RAC OEM is way short of acceptable) the overhead of managing and configuration OEM/Grid is just too high.  And the reason is that the OEM stack is just way too complex a solution for the task at hand.  If you do a opmnctl status you'll see a list of at list eight seperate entities that interact to perform OMS services - and I bet that very few DBAs know what each of them are.   And that's before you add in the agents and internal web apps that perform the various OEM functions.

Compare the OEM/Grid implementation to the far simpler MySQLs LAMP based solution: the number of moving parts in the MySQL soluiton are far less than in OEM, and yet it appears to provide virtually the same functionality (for the DBA at least).

I can't help thinking that in the future every Oracle DBA is going to need to be an expert in the OEM software stack - and consequently an expert in J2EE, Apache, OC4J as well as OEM specifically - to be able to manage enterprise deployments. That's good news for the Oracle DBA job market - as automation reduces the overhead of managing the database, the overhead of managing the automation itself keeps us all in work :-).

 

Comments

[Fog Light]

Good therapy.As well as i read this post.

[HID Bulb]

Well very good post.Thanks for posting this.

[HID Kit]

Can you break down between

echo `find / -name libdb.so.2 -print` >> /etc/ld.so.conf

and

ldconfig -v (run as root)

I dont understand?

[Cheap Computers Canada]

Thanks for the helpful tips they sure will turn out fruitful.

[generic levitra]

is very important to stay informed on these issues of such importance

[records management]

What you did is priceless. Don't hold your IT frustration inside, and let everything out on your blog. I should start up my own blog just for that purpose :)

[Kamagra]

You have become like a guiding light to me. When I started a project on similar topic I could not get enough information to carry further. But as soon as I read your article, my project is doing great and I hope to get an A+ this time. Thanks!

[Kamagra]

You have become like a guiding light to me. When I started a project on similar topic I could not get enough information to carry further. But as soon as I read your article, my project is doing great and I hope to get an A+ this time. Thanks!

[Kamagra]

You have become like a guiding light to me. When I started a project on similar topic I could not get enough information to carry further. But as soon as I read your article, my project is doing great and I hope to get an A+ this time. Thanks!

[Kamagra]

You have become like a guiding light to me. When I started a project on similar topic I could not get enough information to carry further. But as soon as I read your article, my project is doing great and I hope to get an A+ this time. Thanks!

[online cialis]

hey buddy,this is one of the best posts that I�ve ever seen; you may include some more ideas in the same theme. I�m still waiting for some interesting thoughts from your side in your next post.

[Alprazolam]

Really great post, Thank you for sharing This knowledge.Excellently written article, if only all bloggers offered the same level of content as you, the internet would be a much better place. Please keep it up!

[xanax]

Please one more post about that.I wonder how you got so good. This is really a fascinating blog, lots of stuff that I can get into. One thing I just want to say is that your Blog is so perfect

[sildenafil]

That was an interesting piece of information on handwriting analysis. Please post more about graphology. Thank you!