Q: “In one of those perennial round-robins that friends send by e-mail, I found the following: ‘Why do we say something is out of whack? What’s a whack?’ It seems a valid question. Can you supply an answer?”

A: Not with a totally convincing show of certainty, no. But some pointers are possible.
Whack started life in the eighteenth century. It was probably an imitative noise, or perhaps derived from the older thwack, also imitative. The adjective wacky, for somebody or something that is odd, crazy or peculiar (nowadays in a mildly funny way), may come from whack, in that somebody who was crazy behaved as though he had been hit about the head.

The noun developed a number of subsidiary senses. At one time, it could mean a share in a distribution, a portion; this sense was originally thieves’ cant—Francis Grose, in his Dictionary of the Vulgar Tongue of 1785, has “Whack, a share of a booty obtained by fraud” (could physical violence have been involved in some cases?). British English has a couple of phrases that retain that sense. One is pay one’s whack, to pay one’s agreed contribution to shared expenses. Another is top whack, or full whack, for the maximum price or rate for something (“if you go to that shop, you’ll pay top whack”).

There are some other old figurative senses, including a bargain or agreement (which evolved out of the idea of a share), and an attempt at doing something (“I’ll take a whack at that job”). These are mostly American, and it was in the US that the sense you refer to first appeared, in the latter part of the nineteenth century. There seems to have been a phrase in fine whack during that century, meaning that something was in good condition or excellent fettle. (It appears in a letter by John Hay, President Lincoln’s amanuensis, dated August 1863, which describes the President: “The Tycoon is in fine whack. I have rarely seen him more serene and busy. He is managing this war, the draft, foreign relations, and planning a reconstruction of the Union, all at once”.) It doesn’t often turn up in writing, though, so there’s some doubt how widespread it was.
To be out of whack would then have meant the opposite—that something wasn’t on top form or working well. It was first applied to people with ailments (“My back is out of whack”). In the early years of the twentieth century it started to refer to mechanisms. It might be that the sense was influenced by the idea that faulty mechanisms responded to a quick thwack.

From http://www.quinion.com/words/index.php