Keys out of Order Error
apple Verified ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello All,
I think we have a serious problem on our hands. In our lab we have about two dozen macs in many various configurations, but all are running 10.4.2. We have everything from laptops, desktops, even servers. So far half of them have now have suffered the exact same errors: System slow down, Run Disk Utility & get "Keys out of Order" error and fails to rebuild catalog b-tree. Disk Warrior 3.0.3, Disk Utility (tiger), TechTools 4.0.5, Drive Genius, and Norton Utilities (under 10.3.9 out of desperation to recover files) has thus far failed on all accounts to repair all drives. Only Data Rescue II has managed to recover files but not fix the HDs. Since the 10.4.2 update we have had a system go down with this error about 1-2 times per week.
Just last week we lost a 10Terabyte file system on an xServe. NOT FUN. Still trying to recover.
Now for my questions: What do I do? Do I still run 10.4.2? We can't afford to have down time, especially with our servers. Will 10.4.3 fix this problem?
This problem comes at a really unfortunate time. We are currently evaluating Apple in our aging cluster environment to replace about 1000 Dells. With this error Apple is losing out with the people in charge.
-Bao
14 replies
Sign in or join Beachball Help to reply.
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
The keys out of order error usually is the result of improper shut down. Norton Utilities can also cause this problem if used on a 10.4 system. If you run it from a 10.3 system on a 10.4 system it will ruin it! Get rid of it, it is not compatible with 10.4 and support has been dropped for Mac.
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
I've had the same error three times since installing Tiger. I've even tried replacing the drive but it made no difference, the problem still re-occurred after several days. I am not using Norton Utilities or Antivirus. I stay away from those products. When the problem first occurred I bought DiskWarrior, but it was unable to fix it. The only solution I've found is to reformat the drive and do a clean install. However, after 2 or 3 weeks the problem re-occurrs. I've only had this problem since installing Tiger, never previously. I'm currently $300 in the hole because of this and my previously reliable G5 has become a major maintenance task. Not happy.
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello Will.
Keys Out of Order is a major hard drive directory problem which Disk First Aid cannot repair and although Disk Warrior is an excellent disk utility, there are no guarantees DW can repair it.
There is a possibility your new hard drive also has a problem or defect and needs to be returned for an exchange.
As I'm sure you know, it is not a matter of if but when a hard drive has a partial or complete failure since none last forever and new hard drives are not 100% free of defects. I don't believe Tiger is the cause because if so, the same would be reported here in large numbers.
Without warning, I experienced a complete hard drive failure with a new G3 PowerBook less than 1-yr old which Apple replaced at their expense under the standard 1-yr warranty period. OS X was not available yet so I couldn't blame it on that but it certainly would not have caused it.
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Well there are more of us out there with the same problem.
This seems to occur for me when doing software updates. Do the update and the machine won't finish booting up. Boot from the DVD and check the drive....keys out of order and from there on I am dead in the water. reformated the drive and reinstalled. Took me a whole weekend to get it running right again. So this time...no chances...I cloned my drive to another physical drive. Go to do the Quicktime 7.0.2 update and everything runs fine. Go to start up again and oppss...Won't reboot..here we go again, except this time not only was my system hardrive gone bad with the keys error but the clone also goes bad...same error! Except even though the system hard drive gives me the error and won't boot, the files are still accessable?? What gives?? This is getting old FAST!
One of these drives is a year old the other was less than a month old. This can't be bad drives! Two seperate phycical drive both giving the keys out of order error at the same time??Something is up with Tiger!
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
I have a hard time believing that a dozen different machine or so are going down because of hard drive failures all at once. We have performed low level surface scans of all disks and they seem fine. We've tried swapping them out for brand new drives, different brands, etc. Still the problems occurs.
If all it takes is an improper shutdown of the computer to cause this kind of corruption on the hard drive, that's a bad file system. Really Bad! Most recently, (yesterday), I re-formatted, re-zeroed, and re-installed Tiger. After the successful install & reboot it immediately had a corrupted b-tree. I also just performed a surface scan of the drive.
I still say something is up the Tiger 10.4.2!!!
-Bao
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello, Bao and others —
Bao, please accept a warm welcome to Apple Discussions.
I'm concerned that we cannot be responsive or do your problems justice in this forum. I decided to reply primarily to share two thoughts for you to consider. In my opinion:
(1) Your issues are almost certainly not with OS X 10.4.x Tiger, per se. If you'd like to seek specific help in Apple Discussions, you should probably post into the appropriate xServe, xServe RAID, or other <a href="http://discussions.info.apple.com/webx?14@@"">Apple Server Products</a> forum(s). Or perhaps one or more of <a href="http://lists.apple.com/mailman/listinfo">Apple's mailing lists — or any of numerous systems managers forums — would be helpful.
(2) But. There is no "this error" — what you've described is more complex than that. Your lab's IT situation deserves and would benefit greatly from an experienced, in-depth, systematic evaluation — followed by implementation of the identified actions. There's clearly a lot more going on than you've identified — and certainly a lot more than can be addressed in a user-to-user forum.
I'm no expert in servers, networks, or Apple's file management system's inner-workings. But — since you work "in a lab" — I hope you can recognize when a highly complex multivariate problem has been reduced to a single-variable function (using intuition, no less). An equation for disaster — not solutions!
And comparing apples, watermelons, pomegranates — and whales!
Simply take a look through the File Manager Reference and the (now-outdated) Technical Note TN1150: HFS+ Volume Format.
The maladies you've named cry out for consideration of hard-nosed lab-wide IT strategies w.r.t. backup, journaling, optimization, security, use of UPS and other power management, etc. If you're managing files of a size that require (more than?) a 10-Terabyte file system — standard everyday maintenance, optimization, network security, etc. are inadequate. (For example, if your lab is working with huge files, disk optimization is a real issue — though it seldom is for average Mac users. And you're using Norton! Egad!) You need Apple's expertise directly — or a very Mac-experienced and savvy systems troubleshooter — to take a very hard look at your IT practices..
I'm sorry if my message comes across harshly. But these kinds of IT issues underlie any OS or platform. Opining in vague terms in this forum about whether or not OS X 10.4.x does and will meet your needs does a disservice to your lab's realities, in my opinion. Mac OS is "up to the test" — but not if you cheat in day-to-day management . . .
Please forgive the tone of this message — I mean you the best,
Sincerely,
Dean - 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Bao:
Norton Utilities
There's the first culprit I would look at.
B-tree errors & keys out of order amazing coincide with Norton.I apologize for being so basic,
but after upgrading all the Macs, did you repair permissions &
repair the hard drive?Like Dean, I'm wishing you all the best here in finding
your resolution! - 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello.
Point taken. No hard feelings. I really appreciate your point of view and reminding me that I should use the scientific method to try and pin point the exact error. I appoligize for my frustrations and rantings. I will say this: We perform scientific computation and by all respect we are experts in computer software & hardware design. We are one of the flagship showcase for SGI, we also collaborate in many scientific research with several of the National Labs across the country.
For the record, I love Apple. I love their hardware, software, and culture. I have for decades have been trying to convert as many users as possible to Apple.
For the last several months we have been experiencing these problems. I have thought it might be bad installation disks. It's not I have over two dozen different install disks. I thought it might be bad hard drives, memory, power stripes, UPS, computers, ethernet cables, power cables, mice, keyboards, Auto Software update vs. manual down loads, etc. Again, using statistical analysis there seems to be no relations. However, we don't have a large enough sample size to get reasonable statistical power when running a 1/4 factorial analysis or ANOVA analysis. These corruptions primarily happens on desktop & laptops and much less frequently on servers. One thing that does seem common, though we have not been able to fully validate is that corruption seems to occur on restarts. There seems to be a correlation between the number of restarts and corruptions (therefore servers would not see as many corruptions as desktops & laptops). We have started to collect data on this and will be trying to reproduce it. We do run maintenance on our systems. We will also begin collecting error logs and disk IO data to see if we can find a commonalty.
It's just seems too much of a coinidence that after an update to 10.4.2 that all our systems start to get corrupted b-tree catalogs.
Maybe someone let off a small EM Pulse near our lab.
Dean, sorry for using this forum to try and find a solution to a very frustrating problem. We have called and e-mailed Apple technical support with no solution except to re format our drive. But after the second or third time in three weeks on each machine it gets a little old. We do try and maintain our systems to keep them running at their best. (Norton was only used as a last resort in an attempt to recover some of our data). I also understand the frustration that many people out there claim FOUL at the slightest thing that goes wrong. I assure you that I try my best to find the cause before I make a claim. I have defended my doctorate so I do know a little about the scientific method and the difference between cause & effect. I have been a big fan of *nix systems and Mac OS X is one of the Best out there. I have no doubt that in a properly configured & maintained system Mac OS X is "up to the test" as I have setup, benchmarked, tested hundreds of systems. I'm definitely very excited about the enterprise solutions Apple is developing.
Thanks for pointing out that I was being vague. I will be much more detailed in the future.
Sincerely,
Bao - 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Can I say it again,
We don't use Norton! I only used Norton out of desperation after all other tools have failed to recover some of our files.
Just to let you know. We made a bit for bit copy of desktop & laptop Drives before trying the different tools (just in case things get worst!)
I just want to make it clear: Our systems are running for the most part excellent. It's just that under 10.4.2 our re-build rate has gone up significantly, but not to the point that it's crippling us. However, it is annoying.
Sorry I was unclear with my post!
-Bao
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Bao,
Thanks for your thoughtful, considered replies.
Do EMPs tickle?
Pretty-please, don't misunderstand — your posts and queries are very welcome here. Just thought a few suggestions on where to find folks with the most closely similar experience might help. And that glaring at OS X 10.4.x might scatter your focus away from other potential issues.
A touch o' heresy: my years with Macs whisper that "spit, luck, dues paid, and love" may count as much as scientific diagnostics. So open-ended queries and listening closely to ideas that don't seem relevant at first can be helpful. As can experienced, "cold-eyes" evaluation.
We both know that "A happened, then B" is not equivalent to "A caused B" — no matter how attractive the inclination. Yet we've all had experiences to make us ponder this: upgrading and updating OS, using Software Update, adding body English to a jump shot, etc. just can't be coincidental — can they?
I had such an experience about a month ago (see A, B, and C, if you're interested) — and another this week (not posted yet). In both cases, unexpected disk corruption just felt really strange. But in the former case, 'twas in fact bad RAM => directory corruption => worse and worse => no startup. . . and the "co-incidents" were, well, simply adjacent on a timeline.
Almost. Restarts and running Apple installation packages do have something in common: "unseen" diagnostic &/or optimization routines that may suddenly "disrobe the Emperor,"
as it were. Problems mid-accrual suddenly "appear."For the heck of it, I've brainstormed a list of events and circumstances that might cause directory corruption, ordered by their potential for being suspiciously widespread or non-unique. Please note that I created this alone & ~quickly: the list is no doubt incomplete, containing mistaken or poorly expressed entries. Other participants — particularly folks with experience similar to your situation — can surely improve on these ideas. My objective is to catalyze your thinking.
Potential sources of disk corruption:
(a) voltage spikes/surges . . .any high-power electrical devices in the lab or building?
(b) Power interruptions
(c) Improper shutdown — e.g., in reaction to unresponsive system ("freeze")
(d) Pressing "reset"
(e) Kernel panics
(f) System or application crashes
[<small>In the above cases, updated directory information that the Mac OS caches in memory for faster access may never get written to your disk, thus causing directory damage.</small>]
(g) Errors in (an) application's file management, etc. [<small>This may be particularly likely in an environment using custom software: applications perhaps not fully revised for compatibility with OS changes?</small>]
(h) 3rd-party extensions, etc. inadvertently corrupting system data
Use of incompatible utilities/versions; misuse of utilities or Terminal
(j) Any activity during (interrupting) some system functions (including installation, cache cleaning ,some maintenance scripts, etc.)
(k) A program inadvertently writing data to the directory portion of the disk(s)
(l) Improper directory "repair"
(<small>m</small>) HD too full
(n) Huge files, being written-to frequently
(o) Network security problems
(p) Improper software installations — e.g., of OS upgrade
(q) Bad RAM
(r) Bad sectors on startup disk
(s) Defective or failing hard drive.I hope this is of some use to you.
Better living through... cooperation!
Dean - 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Thank you very much for your response. We have been trying to monitor to the best of our abilities anything out of the ordinary. It's kinda funny, we have been developing a method of monitoring metrics in terms of systems to see if we can find correlations for applications such as health & diseases, traffic flow, network computer clusters, etc... We're good enough to get government funding. Since we've had a more than usual number of b-tree corruptions (greater than zero)(16 and counting) we've been trying this method out on these problems.
I understand that the vast majority of users out there are probably not experiencing these problems so it seems reasonable that we have something going on here. On every machine we have ran intensive hardware tests. In one case we have found a bad hard drive. But in all others we can't find a problem. I understand that we might be a special case, but currently many arrows lean toward 10.4.2. That is why we will start monitoring at the system level.
Thank you for the list of potential sources of disk corruption. We have tried to account for most of those items. However, I will try and add the rest of the list into our monitoring database.
Thanks again! This was helpful.
-Bao
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
You're welcome, Bao.
First perusing your thread, I got the idea that you could be facing "undiscovered variables" reponsible for the correlation you've observed. It sounds as if y'all have undertaken what may become a very interesting investigation to isolate and define the phenomenon.
I wish you the best of luck — and hope you'll post back here someday to share what you've discovered, so others can employ the lessons learned.
In the meantime, you probably must also plan for "super-normal" protections for your systems. This may require "custom" systems that aren't widely known or sold "over-the-counter." I hope and trust that you'll find others with similar experiences to share ideas and resources. Toward that end, you may benefit from posting an email address in your profile page.
Warm regards,
Dean</center><hr width=100% color="0066FF"><small>p.s. If/when you do discover reproducible, OS-related "bugs" — you should consider reporting them using BugReporter. (Requires free ADC membership.) You may receive acknowledgment of receipt — and follow-up, as appropriate. There's a BugReporter FAQ that includes, e.g., a link to Apple's Bug Reporting Guide. Also — here's a thorough discussion on How to Report Bugs Effectively. [ <small>Thanks to leigh:) for this link! </small>]</small> - 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
I'm not an expert in so huge IT environments, but what I know is: running Norton is the debut (and the end) of files corruption. And TechTool 4.0.4-5 destroys, sometimes, the boot sector of the hard drives. So, uninstall Norton. Uninstall TechTool agent. Keep only DiskWarrior or Datarescue. At the end of a day, copy some of your finished projects on new harddisks. Erase the used ones. Reformat them. At those dimensions, OS Tiger cannot optimize the files. But copying them (moving them) they will be written in a normal order. That's all I can say.
- 0
Apple Support Community member ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
To complement ~Bee's and Florin's thoughtful comments on this sub-topic:
If you haven't already — noting that you have used DataRescue — you may also want to consider ProSoft's Drive Genius Disk Tools Suite. (AppleLinks review.) I haven't tested it; so, this isn't offered as a recommendation. Others may wish to comment.
I don't want to "beat this" sub-topic "into the ground." Sounds as if you took the risk of using Norton on selected, "floundering" Macs out of desperation (and probably regret mentioning it here). It's so sad that Symantec decided to drop support for a product we once depended on (with earlier Mac OS).
Given what are presumably ~unique data/file management requirements — such as managing huge databases/files that are probably written-to frequently by multiple work stations — these commercial disk management packages may not include all the tools your situation requires. I'm guessing about your requirements, but some of them probably stress file management, disk optimization, etc. . .