Revisited: Installing OEL 5.4 in preparation for Oracle database 11gR2

Linux, Oracle

My OEL 5.3 server with Oracle hasn’t been behaving quite right lately, so all in all, the best choice at this time was to reinstall. At the same time, I wanted to redesign the disk / partition layout.

So here’s a quick recap of this installation.

First, the OS.

  1. Downloaded OEL 5.4 64-bit CD’s 1,2,3,4 and 6 from edelivery.oracle.com
  2. (This server has no DVD player, and previous installations taught me that CD 5 wasn’t necessary for me.)

  3. Booted from first disk, deleted all existing volume groups and selected two (out of six) disks to be available for the OS install.
  4. (Which means Linux will put them together in one Volume Group.)

  5. Disabled SELinux for the time being.
  6. (To prevent some problems when configuring Oracle later, specifically “cannot restore segment prot after reloc” error.)

  7. Created the oracle OS user during OS installation.

That went well enough, so now for some preparations.

  • In one session: Transfer Oracle software to the new server: rsync --progress linux.x64_11gR2_database_?of2.zip oracle@newserver:/src/
  • (Including mkdir /src and making oracle the owner of it.)

  • In a parallell session: Set up Oracle public YUM server, courtesy of
  • http://public-yum.oracle.com/

  • Install required packages: yum install oracle-validated
  • Create OS groups for oracle: usermod -g oinstall -G dba oracle
  • Make sure there is a line in the hosts file for the host itself, so the Oracle installer can map the hostname to an IP.

Create LVM volumes for the remaining disks; here are the commands used:

pvcreate /dev/sdc1 /dev/sdd1
pvcreate /dev/sde1 /dev/sdf1
vgcreate OracleVol01 /dev/sde1 /dev/sdd1
vgcreate OracleVol02 /dev/sdc1 /dev/sdf1
lvcreate -L 100G -n u01 OracleVol01
lvcreate -L 100G -n u02 OracleVol01
lvdisplay
mke2fs -j /dev/OracleVol01/u01
mke2fs -j /dev/OracleVol01/u02
e2label
e2label /dev/OracleVol01/u01 u01
e2label /dev/OracleVol01/u02 u02
vi /etc/fstab
mkdir /u01
mkdir /u02
mount /u01
mount /u02
df -h

And then, install Oracle software, which is another story :)

sh /src/database/runInstaller

How to freeze an EBS instance with no way to recover

E-Business, Oracle

Or: FND_USER table and blocking locks.. I don’t mean recover as in backup, by the way.

Some disclaimers: This scenario happened in an 11.5.10 system, and seems to have been a general problem not specific to our site, but as I haven’t been able to research it further, I don’t know if it could happen elsewhere as well. And even if I couldn’t think of a way around it, some of you might :) Furthermore, the following shouldn’t be a problem for those using the new User Management (UMX) HTML interface to manage EBS users, instead of the older, Forms based “Administer Users” interface. (UMX became available as a patch to 11.5.10, if I remember correctly).

Since the only resolution to this I (and Oracle support) know of, is to shutdown abort the running production EBS database, which is rather grave, I’m hereby noting down our preventive work-around, which banished this issue from our site.

The issue (which happened twice in a couple of weeks):

During normal operation in a non-peak period, typically in the late morning, the Oracle E-Business Suite system would suddenly not accept any more logins. Over the course of perhaps 15 minutes after this, existing users started having trouble accessing different HTML/JSP pages, and after that, the whole system effectively froze up.

The first time this happened, there was no time for troubleshooting, except to discover that there were lots of sessions waiting to get a lock for the FND_USER table, so we had to shut down the services and abort the database. After a call to Oracle support revealed that they did not know of any other way around our situation.

The second time, I was a little more prepared, and though I couldn’t react in time to kill the blocking sessions fast enough the resolve the situation, I did have time to detect a pattern (I’ll get to this in a second).

So the issue in a little more detail, was that – as is normal – each new user session wanted to get hold of their entry in the FND_USER table, to update the TIMESTAMP field. And the already existing sessions also expect to be able to do this once in a while. This was not possible, since one session sat with a blocking lock for it’s own entry. I found this strange, and at least expected to be able to resolve everything by contacting the user to confirm I could kill the session, and by this letting the waiting sessions go on with their business.

But every time the blocker was killed, the next session waiting in line, took over and did not release the lock. And since every user who experienced some kind of hang situation in their session, restarted their browsers and tried to log back in, the waiting line grew much more quickly than it was possible to kill off sessions.

Since there was just a limited amount of time available to experiment with this, I still have no clear understanding of this behaviour. (I had to resort to bouncing/aborting the system at this occasion as well, since that can be done fairly quickly, albeit with the possible necessity of some cleanup work afterwards.)

But the pattern was simply that someone with System Administrator rights had opened the FND_USER entry (in a Form, the correct way) for the SYSADMIN user, for changing its list of Responsibilities. And they had left the Form window open long enough for another user to try the same thing, which was all that’s needed. So if you have a test EBS system you’re not afraid to crash; this could be a useful exercise: Open the SYSADMIN user, change for example an end-date of one of the Responsibilities (back and forth), leave it open, and have another user with Rights attempt to change some other aspect of the SYSADMIN user. If the second session hangs, waiting for the first, have some other users try to log in and out normally, possibly starting up Forms. If you see the above described behaviour, well, then you know it needs to be prevented.

It’s simple enough to avoid, of course. The only policy necessary at our site, instated after the second instance of this issue, was: “Every change to EBS user properties must immediately be followed by closing the form in question”.

As I said, simple enough, and probably very obvious to most EBS sites. But users don’t necessarily think that way, figuring instead that the System is all-powerful and can protect itself against stuff like this. By the way, for those who are wondering, we did also take this opportunity to put in place similar policies for other forms than just the user management. Anyway, the issue never arose again.

Fixing Linux problem: “passwd: Authentication token manipulation error”

Linux, Oracle

I had a very irritating problem on one of my OEL 5 virtual machines, actually on my main Oracle 11g database test server, which is the last stop before production.

Googling did not solve this problem for me, since this error can apparently be caused by several things, so it was unresolved for many weeks, and I considered just creating another virtual machine instead, and reinstalling Oracle.

But this morning, I easily solved the problem, just by reading a man page a little more closely, so I thought I’d share my experience :)

Problem:
For some unknown reason, I could suddenly not change the password of my oracle user, neither as root or as the user itself, because using the command passwd immediately gave me the message “passwd: Authentication token manipulation error”. I could log in as the user, though, for example by doing su – oracle, or by ssh-ing in with public keys set.

Google thought this had to do with a lot of file access or pam settings issues that I knew were not relevant to my problem, partly because I always received this error before I got to write any new password for the user at all.

Solution (for me):
passwd (when run from the root account) also has the option -d, to delete (set to null), a user’s passwd. It occurred to me that this might work, since it might not involve any checks against the existing, somehow corrupt, passwd. And it did. So, for me, the 10 second solution was – as root:


passwd -d oracle
passwd oracle

Hope it might be helpful to others!

OOW09 – second day done

OpenWorld, Oracle

Second day was Monday, which for me started with watching the morning keynotes from the hotel / webcast; the involved persons aren’t the most engaging presenters, especially Charles Phillips. And no really new news either.

Monday turned out to be my big Identity Management day, with sessions on

  • Identity Services in the Cloud
  • Oracle Identity Manager & Oracle Role Manager 11g
  • Oracle Access Manager 11g

One good Hands-on-Lab as well, on a product I’ve been following for a while, the Oracle E-Business Suite Application Change Management Pack (ACP). I know of several E-Business customers back home in Norway who don’t really have good control over their customizations.

It would probably pay back quickly for a lot of these, to package their customizations as standard patches, and optionally manage them with the builtin ACP patch manager, integrated with a source control system.

A couple of sessions I’ve noted down for flipping through later:

S308223 DBA ASM
S312645 Database performance on Linux
S308361 High performance PL/SQL
S312448 + S312449 SQL Developer
S312693 Database 11g in virtualized storage

And I met Marius Ciortea from Oracle (who works with Enterprise Social Networking) in the OTN Lounge, and I agreed (not naively, I hope) to a short attendee interview for the live OTN feed on

http://www.oracle.com/us/openworld/034626.htm

OOW09 – first day done

OpenWorld, Oracle

A busy first day is done. I especially liked the session Linux Shell Scripting Craftsmanship, partly because it’s interesting subject with lots of possiblilities for improvement (no; read: many things done wrong). But mainly because Ray Smith was a great presenter, so thanks, very good presentation. I see from the net that he’s gotten much praise for his presentation from other conferences, by the way.

On the subject of shell script coding: The propositions and guidelines from this session would be a major contribution to most coding projects I know of, so please look up the slides when they appear online, they are really good value if you’re a developer, DBS or sysadmin. Session ID S312131.

I also enjoyed going through the OPSS session; “Developing Secure Applications with Oracle Platform Security Services” – S310003

In the evening, Oracle Norway arranged dinner for all Norwegian customers and partners at the Stinking Rose, which means I won’t be very popular as a session seat neighbour today, since they serve everything with very large amounts of garlic.

An interesting side note from the dinner: We are 111 attendees from Norway, which is very good. But this is more than both Sweden and Denmark, which are both larger countries, and larger Oracle markets (esp. Sweden), so it’s actually very good.

Realtime: Now the keynotes are finished for the morning, so it’s time to head over to Moscone West.

Bloggers physically present at OpenWorld 2009 list

Oracle

13.10.2009:
Updated with latest confirmed attendees of the Bloggers Meetup tonight. Marcel Kratochvil, Jacco Landlust, William Vambenepe, Stuart Marks, Avrom I. Roy-Faderman, Stanley ACE Director, OCP Advisor

12.10.2009:
There are always more to add, so please check back (and give feedback; I’ve received most via Twitter – eespenh – and that’s great). Since last, I’ve added Daniel Morgan, Karen Morton (because I noticed them standing in the Unconference area), Fenng, Khanderao, Merv Adrian.

11.10.2009:
Since yesterday here at OpenWorld, I noticed that Bex Huff, Hans Forbrich, Moans Nogood (not really) and Jason Jones are also here. And Claudia Zeiler (girlgeek), whom I just met outside the OTN lounge. Wonder if I’ve got half of everyone before the week’s over?

http://avromroyfaderman.com – Avrom I. Roy-Faderman
http://badgerworks.org/ – Ted Simpson
http://bexhuff.com/ – Bex Huff
http://blog.arungupta.me/ – Arun Gupta
http://blog.dbvisit.com/ – Arjen Visser
http://blog.rayapps.com/ – Raimonds Simanovskis
http://blog.thebdgway.com/ – Chris Bucchere
http://carymillsap.blogspot.com/ – Cary Millsap
http://cheeseheaddba.blogspot.com/ – Todd Sheetz
http://daust.blogspot.com/ – Dietmar Aust
http://dbanotes.net/ – Fenng
http://debrasoracle.blogspot.com/ – Debra Lilley
http://dgielis.blogspot.com/ – Dimitri Gielis
http://dpeake.blogspot.com/ – David Peake
http://ebizocp.blogspot.com – OCP Advisor
http://eisele.net/blog/ – Markus Eisele
http://empoprise-bi.blogspot.com/ – John E. Bredehoft
http://eternal-donut.blogspot.com – Marcel Kratochvil
http://fuadarshad.net/wp/ – Fuad Arshad
http://fusioninsidersedition.blogspot.com/ – George J. Trujillo, Jr.
https://fusiontech.wordpress.com/ – Espen Barroso-Gomez – myself
http://garethroberts.blogspot.com/ – Gareth Roberts
http://girlgeek-oracle.blogspot.com/ – Claudia Zeiler
http://gvwoods.blogspot.com/ – George Woods
http://hansforbrich.blogspot.com/ – Hans Forbrich
http://jarneil.wordpress.com/ – Jason Arneil
http://java.net/blog/108 – Kohsuke Kawaguchi
http://jonathanlewis.wordpress.com – Jonathan Lewis
http://karenmorton.blogspot.com/ – Karen Morton
http://khanderaotech.blogspot.com/ – Khanderao
http://matttopper.com/ – Matt Topper
http://mervadrian.wordpress.com/ – Merv Adrian
http://michelledbaunleashed.blogspot.com/ – Michelle Malcher
http://newappsdba.blogspot.com – Dave B
http://oracle-base.com/blog/ – Tim Hall
http://oracle-internals.com – Jonah H. Harris
http://oracledoug.com/serendipity – Doug Burns
http://oraclemva.wordpress.com – Jacco Landlust
http://oraclestorageguy.typepad.com/oraclestorageguy/ – Jeff Browning
http://orclville.blogspot.com/ – Floyd Teter
http://prodlife.wordpress.com/ – Gwen Shapira
http://realjavasoa.blogspot.com/ – Jason Jones
http://radiofreetooting.blogspot.com/ – Andrew Clarke
http://richardfoote.wordpress.com/ – Richard Foote
http://roelhartman.blogspot.com/ – Roel Hartman
http://rwijk.blogspot.com/ – Rob Van Wijk
http://stage.vambenepe.com – William Vambenepe
http://stuartmarks.wordpress.com – Stuart Marks
http://sysdba.wordpress.com/ – Lutz Hartmann
http://tbray.org/ongoing/ – Tim Bray
http://technology.amis.nl/blog/6336 – Lucas Jellema
http://tkyte.blogspot.com/ – Tom Kyte
http://tonguc.wordpress.com/ – H. Tonguç Yilmaz
http://wedonotuse.blogspot.com/ – Moans Nogood
http://www.liberidu.com/blog/?p=1281 – Marco Gralike
http://www.morganslibrary.org/blog.html – Daniel Morgan
http://www.ome-b.nl – Douwe Pieter van den Bos
http://www.pythian.com/news/author/alex – Alex Gorbachev
http://www.wtfistheacevest.com – Stanley ACE Director

Others who might be here or have more than one main contributor:

http://blog.softwareinsider.org – R “Ray” Wang
http://oraganism.wordpress.com/
http://oracle-magic.blogspot.com/
http://talentedapps.wordpress.com – Mark Bennett, Meg Bear, Amy Wilson

OpenWorld 2009 – who’s there?

Oracle

I’ll be going over there myself in a few hours, crossing the nine hour time difference. But I started wondering which of my fellow, admittedly much more industrious, blog writers, would be present at Oracle OpenWorld 2009 (#oow09) as well. I’ve read their notices, but my memory isn’t good enough to piece that into a picture.

So here’s a first attempt to list those blogs whose author has expressed an intention to attend OpenWorld this year. This is just the result of one hour of searching through OraNA.info and Google, and not counting blogs under blog.oracle.com. Some blogs I’ve left out because it’s unclear to me which one person represents it best; please leave a comment in cases where I’ve gotten things wrong, though! As an example, blogs.ioug.org isn’t really Judi Doolittles personal blog, even if she’s responsible for the content, but she might have wanted it in such a list anyway?

So I’m just making up the rules as I go, I guess (why? because it’s two o’clock in the morning (the taxi for the airport arrives in two hours) :)
I’m probably missing a lot of persons in this first attempt, anyway.

Digression:
The number of people attending OOW according to the event (created by Ontario Emperor) in Oracle Community (oraclecommunity.net) at present is 7. So if you’re a member of Oracle Community, do sign up for the event, and if not, do sign up for Oracle Community. And Oracle Mix ;) (But Oracle Mix won’t let me in this week; that’s another story, I guess.)

Another digression:
The number of bloggers attending the Blogger Meetup event (which is Tuesday at 6pm at Jillians, by the way; look it up, is around three times as many. Looking forward to meeting many of you there.

So this is the list at the moment:

http://avromroyfaderman.com – Avrom I. Roy-Faderman
http://badgerworks.org/ – Ted Simpson
http://bexhuff.com/ – Bex Huff
http://blog.arungupta.me/ – Arun Gupta
http://blog.dbvisit.com/ – Arjen Visser
http://blog.rayapps.com/ – Raimonds Simanovskis
http://blog.thebdgway.com/ – Chris Bucchere
http://carymillsap.blogspot.com/ – Cary Millsap
http://cheeseheaddba.blogspot.com/ – Todd Sheetz
http://daust.blogspot.com/ – Dietmar Aust
http://debrasoracle.blogspot.com/ – Debra Lilley
http://dbanotes.net/ – Fenng
http://dgielis.blogspot.com/ – Dimitri Gielis
http://dpeake.blogspot.com/ – David Peake
http://ebizocp.blogspot.com – OCP Advisor
http://eisele.net/blog/ – Markus Eisele
http://empoprise-bi.blogspot.com/ – John E. Bredehoft
http://eternal-donut.blogspot.com – Marcel Kratochvil
http://fuadarshad.net/wp/ – Fuad Arshad
http://fusioninsidersedition.blogspot.com/ – George J. Trujillo, Jr.
https://fusiontech.wordpress.com/ – Espen Barroso-Gomez – myself
http://garethroberts.blogspot.com/ – Gareth Roberts
http://girlgeek-oracle.blogspot.com/ – Claudia Zeiler
http://gvwoods.blogspot.com/ – George Woods
http://hansforbrich.blogspot.com/ – Hans Forbrich
http://jarneil.wordpress.com/ – Jason Arneil
http://java.net/blog/108 – Kohsuke Kawaguchi
http://jonathanlewis.wordpress.com – Jonathan Lewis
http://karenmorton.blogspot.com/ – Karen Morton
http://khanderaotech.blogspot.com/ – Khanderao
http://matttopper.com/ – Matt Topper
http://mervadrian.wordpress.com/ – Merv Adrian
http://michelledbaunleashed.blogspot.com/ – Michelle Malcher
http://newappsdba.blogspot.com/ – Dave B
http://oracle-base.com/blog/ – Tim Hall
http://oracle-internals.com/ – Jonah H. Harris
http://oracledoug.com/serendipity/ – Doug Burns
http://oraclemva.wordpress.com – Jacco Landlust
http://oraclestorageguy.typepad.com/oraclestorageguy/ – Jeff Browning
http://orclville.blogspot.com/ – Floyd Teter
http://prodlife.wordpress.com/ – Gwen Shapira
http://realjavasoa.blogspot.com/ – Jason Jones
http://radiofreetooting.blogspot.com/ – Andrew Clarke
http://richardfoote.wordpress.com/ – Richard Foote
http://roelhartman.blogspot.com/ – Roel Hartman
http://rwijk.blogspot.com/ – Rob Van Wijk
http://stage.vambenepe.com – William Vambenepe
http://stuartmarks.wordpress.com – Stuart Marks
http://sysdba.wordpress.com/ – Lutz Hartmann
http://tbray.org/ongoing/ – Tim Bray
http://technology.amis.nl/blog/6336 – Lucas Jellema
http://tkyte.blogspot.com/ – Tom Kyte
http://tonguc.wordpress.com/ – H. Tonguç Yilmaz
http://wedonotuse.blogspot.com/ – Moans Nogood
http://www.liberidu.com/blog/?p=1281 – Marco Gralike
http://www.ome-b.nl – Douwe Pieter van den Bos
http://www.morganslibrary.org/blog.html – Daniel Morgan
http://www.pythian.com/news/author/alex – Alex Gorbachev
http://www.wtfistheacevest.com – Stanley ACE Director

Others who might be here or have more than one main contributor:

http://blog.softwareinsider.org – R “Ray” Wang
http://blogs.ioug.org – IOUG / Judi Doolittle, Steve Lemme
http://blogs.sun.com/projectkenai – Sharat Chander, JB
http://oraclebiwasig.blogspot.com – Shyam Varan Nath, John Haydu
http://oraganism.wordpress.com – Martin Nash, Neil Johnson, Pawel Krol
http://talentedapps.wordpress.com – Mark Bennett, Meg Bear, Amy Wilson

Installing 11gR2 on OEL 5.3 – postrequisites

Oracle

A quick followup on my post on installing 11gR2 a little while ago (last post).

There were actually only two things I had problems with after this Oracle install on OEL 5.3:

  1. I couldn’t start the listener (got an error like “cannot restore segment prot after reloc: Permission denied”) until I had set SELinux to Permissive:
    setenforce 0
    (See for example http://beyondoracle.wordpress.com/2009/03/05/rhel5-or-oel5-oracle-10g11g-error-while-loading-shared-libraries-or-stalled-oui/ for a discussion on this)
  2. I couldn’t reach the administrative URL’s before I had configured/disabled the local firewall, for this test instance with the quick fix:
    service iptables stop