格致

Followup to:  The Moral Void

A widespread excuse for avoiding rationality is the widespread belief that it is "rational" to believe life is meaningless, and thus suffer existential angst.  This is one of the secondary reasons why it is worth discussing the nature of morality.  But it's also worth attacking existential angst directly.

I suspect that most existential angst is not really existential.  I think that most of what is labeled "existential angst" comes from trying to solve the wrong problem.

Let's say you're trapped in an unsatisfying relationship, so you're unhappy.  You consider going on a skiing trip, or you actually go on a skiing trip, and you're still unhappy.  You eat some chocolate, but you're still unhappy.  You do some volunteer work at a charity (or better yet, work the same hours professionally and donate the money, thus applying the Law of Comparative Advantage) and you're still unhappy because you're in an unsatisfying relationship.

So you say something like:  "Skiing is meaningless, chocolate is meaningless, charity is meaningless, life is doomed to be an endless stream of woe."  And you blame this on the universe being a mere dance of atoms, empty of meaning.  Not necessarily because of some kind of subconsciously deliberate Freudian substitution to avoid acknowledging your real problem, but because you've stopped hoping that your real problem is solvable.  And so, as a sheer unexplained background fact, you observe that you're always unhappy.

Maybe you're poor, and so always unhappy.  Nothing you do solves your poverty, so it starts to seem like a universal background fact, along with your unhappiness.  So when you observe that you're always unhappy, you blame this on the universe being a mere dance of atoms.  Not as some kind of Freudian substitution, but because it has ceased to occur to you that there does exist some possible state of affairs in which life is not painful.

What about rich heiresses with everything in the world available to buy, who still feel unhappy?  Perhaps they can't get themselves into satisfying romantic relationships.  One way or another, they don't know how to use their money to create happiness - they lack the expertise in hedonic psychology and/or self-awareness and/or simple competence.

So they're constantly unhappy - and they blame it on existential angst, because they've already solved the only problem they know how to solve.  They already have enough money and they've already bought all the toys.  Clearly, if there's still a problem, it's because life is meaningless.

If someone who weighs 560 pounds suffers from "existential angst", allegedly because the universe is a mere dance of particles, then stomach reduction surgery might drastically change their views of the metaphysics of morality.

I'm not a fan of Timothy Ferris, but The Four-Hour Workweek does make an interesting fun-theoretic observation:

Let's assume we have 10 goals and we achieve them - what is the desired outcome that makes all the effort worthwhile?  The most common response is what I also would have suggested five years ago: happiness.  I no longer believe this is a good answer. Happiness can be bought with a bottle of wine and has become ambiguous through overuse.  There is a more precise alternative that reflects what I believe the actual objective is.

Bear with me.  What is the opposite of happiness? Sadness?  No.  Just as love and hate are two sides of the same coin, so are happiness and sadness.  Crying out of happiness is a perfect illustration of this.  The opposite of love is indifference, and the opposite of happiness is - here's the clincher - boredom.

 Excitement is the more practical synonym for happiness, and it is precisely what you should strive to chase.  It is the cure-all. When people suggest you follow your "passion" or your "bliss," I propose that they are, in fact, referring to the same singular concept: excitement.

This brings us full circle.  The question you should be asking isn't "What do I want?" or "What are my goals?" but "What would excite me?"

Remember - boredom is the enemy, not some abstract "failure."

 Living like a millionaire requires doing interesting things and not just owning enviable things.

I don't endorse all of the above, of course.  But note the SolvingTheWrongProblem anti-pattern Ferris describes.  It was on reading the above that I first generalized ExistentialAngstFactory.

Now, if someone is in a unproblematic, loving relationship; and they have enough money; and no major health problems; and they're signed up for cryonics so death is not approaching inexorably; and they're doing exciting work that they enjoy; and they believe they're having a positive effect on the world...

...and they're still unhappy because it seems to them that the universe is a mere dance of atoms empty of meaning, then we may have a legitimate problem here.  One that, perhaps, can only be resolved by a very long discussion of the nature of morality and how it fits into a reductionist universe.  (In progress.)

But, mostly, I suspect that when people complain about the empty meaningless void, it is because they have at least one problem that they aren't thinking about solving - perhaps because they never identified it.  Being able to identify your own problems is a feat of rationality that schools don't explicitly train you to perform.  And they haven't even been told that an un-focused-on problem might be the source of their "existential angst" - they've just been told to blame it on existential angst.

That's the other reason it might be helpful to understand the nature of morality - even if it just adds up to moral normality - because it tells you that if you're constantly unhappy, it's not because the universe is empty of meaning.

Or maybe believing the universe is a "mere dance of particles" is one more factor contributing to human unhappiness; in which case, again, people can benefit from eliminating that factor.

If it seems to you like nothing you do makes you happy, and you can't even imagine what would make you happy, it's not because the universe is made of particle fields.  It's because you're still solving the wrong problem.  Keep searching, until you find the visualizable state of affairs in which the existential angst seems like it should go away - that might (or might not) tell you the real problem; but at least, don't blame it on reductionism.

Added:  Several commenters pointed out that random acts of brain chemistry may also be responsible for depression, even if your life is otherwise fine.  As far as I know, this is true.  But, once again, it won't help to mistake that random act of brain chemistry as being about existential issues; that might prevent you from trying neuropharmaceutical interventions.

Most people want to succeed, but most also have moral qualms about doing whatever it takes.  People with unusually strong ambitions or weak qualms, however, should be willing to do much more, even murder.  And at the top of each walk of life we expect to find a disproportionate fraction not only of high ability folks, but also of high ambition and low qualm folks. 

We thus naturally worry about finding the darkest forms of foul play at the top.  Literature is full of plausible-seeming scenarios where by leaders in government, business, and even the arts commit the most terrible crimes to get ahead.  But we tend to believe these stories more about leaders long ago or far away, and less about leaders in admirable walks of life, like religion, academia, or the arts.  Is this just wishful thinking, or is there more to it?

An interesting concrete example is corporate assassins.  We hear of assassination of leaders in crime or politics, at least far away, but less often in business.  Given how little it seems to cost to have someone killed, why don't more corporations have their competitors' leaders knocked off? 

Followup to:  Where Recursive Justification Hits Bottom, Rebelling Within Nature

Years ago, Eliezer1999 was convinced that he knew nothing about morality.

For all he knew, morality could require the extermination of the human species; and if so he saw no virtue in taking a stand against morality, because he thought that, by definition, if he postulated that moral fact, that meant human extinction was what "should" be done.

I thought I could figure out what was right, perhaps, given enough reasoning time and enough facts, but that I currently had no information about it.  I could not trust evolution which had built me.  What foundation did that leave on which to stand?

Well, indeed Eliezer1999 was massively mistaken about the nature of morality, so far as his explicitly represented philosophy went.

But as Davidson once observed, if you believe that "beavers" live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false.  You must get at least some of your beliefs right, before the remaining ones can be wrong about anything.

My belief that I had no information about morality was not internally consistent.

Saying that I knew nothing felt virtuous, for I had once been taught that it was virtuous to confess my ignorance.  "The only thing I know is that I know nothing," and all that.  But in this case I would have been better off considering the admittedly exaggerated saying, "The greatest fool is the one who is not aware they are wise."  (This is nowhere near the greatest kind of foolishness, but it is a kind of foolishness.)

Was it wrong to kill people?  Well, I thought so, but I wasn't sure; maybe it was right to kill people, though that seemed less likely.

What kind of procedure would answer whether it was right to kill people?  I didn't know that either, but I thought that if you built a generic superintelligence (what I would later label a "ghost of perfect emptiness") then it could, you know, reason about what was likely to be right and wrong; and since it was superintelligent, it was bound to come up with the right answer.

The problem that I somehow managed not to think too hard about, was where the superintelligence would get the procedure that discovered the procedure that discovered the procedure that discovered morality - if I couldn't write it into the start state that wrote the successor AI that wrote the successor AI.

As Marcello Herreshoff later put it, "We never bother running a computer program unless we don't know the output and we know an important fact about the output."  If I knew nothing about morality, and did not even claim to know the nature of morality, then how could I construct any computer program whatsoever - even a "superintelligent" one or a "self-improving" one - and claim that it would output something called "morality"?

There are no-free-lunch theorems in computer science - in a maxentropy universe, no plan is better on average than any other.  If you have no knowledge at all about "morality", there's also no computational procedure that will seem more likely than others to compute "morality", and no meta-procedure that's more likely than others to produce a procedure that computes "morality".

I thought that surely even a ghost of perfect emptiness, finding that it knew nothing of morality, would see a moral imperative to think about morality.

But the difficulty lies in the word think.  Thinking is not an activity that a ghost of perfect emptiness is automatically able to carry out.  Thinking requires running some specific computation that is the thought.  For a reflective AI to decide to think, requires that it know some computation which it believes is more likely to tell it what it wants to know, than consulting an Ouija board; the AI must also have a notion of how to interpret the output.

If one knows nothing about morality, what does the word "should" mean, at all?  If you don't know whether death is right or wrong - and don't know how you can discover whether death is right or wrong - and don't know whether any given procedure might output the procedure for saying whether death is right or wrong - then what do these words, "right" and "wrong", even mean?

If the words "right" and "wrong" have nothing baked into them - no starting point - if everything about morality is up for grabs, not just the content but the structure and the starting point and the determination procedure - then what is their meaning?  What distinguishes, "I don't know what is right" from "I don't know what is wakalixes"?

A scientist may say that everything is up for grabs in science, since any theory may be disproven; but then they have some idea of what would count as evidence that could disprove the theory.  Could there be something that would change what a scientist regarded as evidence?

Well, yes, in fact; a scientist who read some Karl Popper and thought they knew what "evidence" meant, could be presented with the coherence and uniqueness proofs underlying Bayesian probability, and that might change their definition of evidence.  They might not have had any explicit notion, in advance, that such a proof could exist.  But they would have had an implicit notion.  It would have been baked into their brains, if not explicitly represented therein, that such-and-such an argument would in fact persuade them that Bayesian probability gave a better definition of "evidence" than the one they had been using.

In the same way, you could say, "I don't know what morality is, but I'll know it when I see it," and make sense.

But then you are not rebelling completely against your own evolved nature.  You are supposing that whatever has been baked into you to recognize "morality", is, if not absolutely trustworthy, then at least your initial condition with which you start debating.  Can you trust your moral intuitions to give you any information about morality at all, when they are the product of mere evolution?

But if you discard every procedure that evolution gave you and all its products, then you discard your whole brain.  You discard everything that could potentially recognize morality when it sees it.  You discard everything that could potentially respond to moral arguments by updating your morality.  You even unwind past the unwinder: you discard the intuitions underlying your conclusion that you can't trust evolution to be moral.  It is your existing moral intuitions that tell you that evolution doesn't seem like a very good source of morality.  What, then, will the words "right" and "should" and "better" even mean?

Humans do not perfectly recognize truth when they see it, and hunter-gatherers do not have an explicit concept of the Bayesian criterion of evidence.  But all our science and all our probability theory was built on top of a chain of appeals to our instinctive notion of "truth".  Had this core been flawed, there would have been nothing we could do in principle to arrive at the present notion of science; the notion of science would have just sounded completely unappealing and pointless.

One of the arguments that might have shaken my teenage self out of his mistake, if I could have gone back in time to argue with him, was the question:

Could there be some morality, some given rightness or wrongness, that human beings do not perceive, do not want to perceive, will not see any appealing moral argument for adopting, nor any moral argument for adopting a procedure that adopts it, etcetera?  Could there be a morality, and ourselves utterly outside its frame of reference?  But then what makes this thing morality - rather than a stone tablet somewhere with the words 'Thou shalt murder' written on them, with absolutely no justification offered?

So all this suggests that you should be willing to accept that you might know a little about morality.  Nothing unquestionable, perhaps, but an initial state with which to start questioning yourself.  Baked into your brain but not explicitly known to you, perhaps; but still, that which your brain would recognize as right is what you are talking about.  You will accept at least enough of the way you respond to moral arguments as a starting point, to identify "morality" as something to think about.

But that's a rather large step.

It implies accepting your own mind as identifying a moral frame of reference, rather than all morality being a great light shining from beyond (that in principle you might not be able to perceive at all).  It implies accepting that even if there were a light and your brain decided to recognize it as "morality", it would still be your own brain that recognized it, and you would not have evaded causal responsibility - or evaded moral responsibility either, on my view.

It implies dropping the notion that a ghost of perfect emptiness will necessarily agree with you, because the ghost might occupy a different moral frame of reference, respond to different arguments, be asking a different question when it computes what-to-do-next.

And if you're willing to bake at least a few things into the very meaning of this topic of "morality", this quality of rightness that you are talking about when you talk about "rightness" - if you're willing to accept even that morality is what you argue about when you argue about "morality" - then why not accept other intuitions, other pieces of yourself, into the starting point as well?

Why not accept that, ceteris paribus, joy is preferable to sorrow?

You might later find some ground within yourself or built upon yourself with which to criticize this - but why not accept it for now?  Not just as a personal preference, mind you; but as something baked into the question you ask when you ask "What is truly right"?

But then you might find that you know rather a lot about morality!  Nothing certain - nothing unquestionable - nothing unarguable - but still, quite a bit of information.  Are you willing to relinquish your Socratean ignorance?

I don't argue by definitions, of course.  But if you claim to know nothing at all about morality, then you will have problems with the meaning of your words, not just their plausibility.

Today begins a conference on Global Catastrophic Risk I'm attending.  Coincidentally, Bryan Caplan recently pointed us to a new paper by Andrew Healy:

Using comprehensive data on natural disasters, government spending, and election returns, I show that voters reward disaster relief spending but not disaster prevention spending. This aspect of voter behavior creates a large distortion in the incentives that governments face, since the data show that prevention spending substantially reduces future damage. ... Given mean annual prevention spending of $195 million and mean disaster damage of $16.5 billion, the regression estimates that a $1 increase in prevention spending resulted in a $8.30 decrease in disaster damage, and this estimate captures only benefits that occur in the five years from 2000-2004.

My and other conference presentation describe other disaster biases, such as not paying due attention to the very largest possible disasters, and not make extra preparations against human extinction. 

Followup to:  Thou Art Godshatter, Joy in the Merely Real, Is Morality Given?, Rebelling Within Nature 

How, oh how, did an unloving and mindless universe, cough up minds who were capable of love?

"No mystery in that," you say, "it's just a matter of natural selection."

But natural selection is cruel, bloody, and bloody stupid.  Even when, on the surface of things, biological organisms aren't directly fighting each other - aren't directly tearing at each other with claws - there's still a deeper competition going on between the genes.  Genetic information is created when genes increase their relative frequency in the next generation - what matters for "genetic fitness" is not how many children you have, but that you have more children than others.  It is quite possible for a species to evolve to extinction, if the winning genes are playing negative-sum games.

How, oh how, could such a process create beings capable of love?

"No mystery," you say, "there is never any mystery-in-the-world; mystery is a property of questions, not answers.  A mother's children share her genes, so the mother loves her children."

But sometimes mothers adopt children, and still love them.  And mothers love their children for themselves, not for their genes.

"No mystery," you say, "Individual organisms are adaptation-executers, not fitness-maximizers.   Evolutionary psychology is not about deliberately maximizing fitness - through most of human history, we didn't know genes existed.  We don't calculate our acts' effect on genetic fitness consciously, or even subconsciously."

But human beings form friendships even with non-relatives: how, oh how, can it be?

"No mystery, for hunter-gatherers often play Iterated Prisoner's Dilemmas, the solution to which is reciprocal altruism.  Sometimes the most dangerous human in the tribe is not the strongest, the prettiest, or even the smartest, but the one who has the most allies."

Yet not all friends are fair-weather friends; we have a concept of true friendship - and some people have sacrificed their life for their friends.  Would not such a devotion tend to remove itself from the gene pool?

"You said it yourself: we have a concept of true friendship and fair-weather friendship.  We can tell, or try to tell, the difference between someone who considers us a valuable ally, and someone executing the friendship adaptation.  We wouldn't be true friends with someone who we didn't think was a true friend to us - and someone with many true friends is far more formidable than someone with many fair-weather allies."

And Mohandas Gandhi, who really did turn the other cheek?  Those who try to serve all humanity, whether or not all humanity serves them in turn?

"That perhaps is a more complicated story.  Human beings are not just social animals.  We are political animals who argue linguistically about policy in adaptive tribal contexts.  Sometimes the formidable human is not the strongest, but the one who can most skillfully argue that their preferred policies match the preferences of others."

Um... that doesn't explain Gandhi, or am I missing something?

"The point is that we have the ability to argue about 'What should be done?' as a proposition - we can make those arguments and respond to those arguments, without which politics could not take place."

Okay, but Gandhi?

"Believed certain complicated propositions about 'What should be done?' and did them."

That sounds like it could explain any possible human behavior.

"If we traced back the chain of causality through all the arguments, it would involve: a moral architecture that had the ability to argue general abstract moral propositions like 'What should be done to people?'; appeal to hardwired intuitions like fairness, a concept of duty, pain aversion + empathy; something like a preference for simple moral propositions, probably reused from our previous Occam prior; and the end result of all this, plus perhaps memetic selection effects, was 'You should not hurt people' in full generality -"

And that gets you Gandhi.

"Unless you think it was magic, it has to fit into the lawful causal development of the universe somehow."

Well... I certainly won't postulate magic, under any name.

"Good."

But come on... doesn't it seem a little... amazing... that hundreds of millions of years worth of evolution's death tournament could cough up mothers and fathers, sisters and brothers, husbands and wives, steadfast friends and honorable enemies, true altruists and guardians of causes, police officers and loyal defenders, even artists sacrificing themselves for their art, all practicing so many kinds of love?  For so many things other than genes?  Doing their part to make their world less ugly, something besides a sea of blood and violence and mindless replication?

"Are you claiming to be surprised by this?  If so, question your underlying model, for it has led you to be surprised by the true state of affairs.  Since the beginning, not one unusual thing has ever happened."

But how is it not surprising?

"What are you suggesting, that some sort of shadowy figure stood behind the scenes and directed evolution?"

Hell no.  But -

"Because if you were suggesting that, I would have to ask how that shadowy figure originally decided that love was a desirable outcome of evolution.  I would have to ask where that figure got preferences that included things like love, friendship, loyalty, fairness, honor, romance, and so on.  On evolutionary psychology, we can see how that specific outcome came about - how those particular goals rather than others were generated in the first place.  You can call it 'surprising' all you like.  But when you really do understand evolutionary psychology, you can see how parental love and romance and honor, and even true altruism and moral arguments, bear the specific design signature of natural selection in particular adaptive contexts of the hunter-gatherer savanna.  So if there was a shadowy figure, it must itself have evolved - and that obviates the whole point of postulating it."

I'm not postulating a shadowy figure!  I'm just asking how human beings ended up so nice.

"Nice!  Have you looked at this planet lately?  We also bear all those other emotions that evolved, too - which would tell you very well that we evolved, should you begin to doubt it.  Humans aren't always nice."

We're one hell of a lot nicer than the process that produced us, which lets elephants starve to death when they run out of teeth, and doesn't anesthetize a gazelle even as it lays dying and is of no further importance to evolution one way or the other.  It doesn't take much to be nicer than evolution.  To have the theoretical capacity to make one single gesture of mercy, to feel a single twinge of empathy, is to be nicer than evolution.  How did evolution, which is itself so uncaring, create minds on that qualitatively higher moral level than itself?  How did evolution, which is so ugly, end up doing anything so beautiful?

"Beautiful, you say?  Bach's Little Fugue in G Minor may be beautiful, but the sound waves, as they travel through the air, are not stamped with tiny tags to specify their beauty.  If you wish to find explicitly encoded a measure of the fugue's beauty, you will have to look at a human brain - nowhere else in the universe will you find it.  Not upon the seas or the mountains will you find such judgments written: they are not minds, they cannot think."

Perhaps that is so, but still I ask:  How did evolution end up doing anything so beautiful, as giving us the ability to admire the beauty of a flower?

"Can you not see the circularity in your question?  If beauty were like some great light in the sky that shined from outside humans, then your question might make sense - though there would still be the question of how humans came to perceive that light.  You evolved with a psychology unlike evolution:  Evolution has nothing like the intelligence or the precision required to exactly quine its goal system.  In coughing up the first true minds, evolution's simple fitness criterion shattered into a thousand values.  You evolved with a psychology that attaches utility to things which evolution does not care about, like human life and happiness.  And then you look back and say, 'How marvelous, that uncaring evolution produced minds that care about sentient life!'  So your great marvel and wonder, that seems like far too much coincidence, is really no coincidence at all."

But then it is still amazing that this particular circular loop, happened to loop around such important things as beauty and altruism.

"I don't think you're following me here.  To you, it seems natural to privilege the beauty and altruism as special, as preferred, because you value them highly; and you don't see this as a unusual fact about yourself, because many of your friends do likewise.  So you expect that a ghost of perfect emptiness would also value life and happiness - and then, from this standpoint outside reality, a great coincidence would indeed have occurred."

But you can make arguments for the importance of beauty and altruism from first principles - that our aesthetic senses lead us to create new complexity, instead of repeating the same things over and over; and that altruism is important because it takes us outside ourselves, gives our life a higher meaning than sheer brute selfishness.

"Oh, and that argument is going to move even a ghost of perfect emptiness - now that you've appealed to slightly different values?  Those aren't first principles, they're just different principles.  Even if you've adopted a high-falutin' philosophical tone, still there are no universally compelling arguments.  All you've done is pass the recursive buck."

You don't think that, somehow, we evolved to tap into something beyond -

"What good does it do to suppose something beyond?  Why should we pay more attention to that beyond thing, than we pay to our existence as humans?  How does it alter your personal responsibility, to say that you were only following the orders of the beyond thing?  And you would still have evolved to let the beyond thing, rather than something else, direct your actions.  You are only passing the recursive buck.  Above all, it would be too much coincidence."

Too much coincidence?

"A flower is beautiful, you say.  Do you think there is no story behind that beauty, or that science does not know the story?  Flower pollen is transmitted by bees, so by sexual selection, flowers evolved to attract bees - by imitating certain mating signs of bees, as it happened; the flowers' patterns would look more intricate, if you could see in the ultraviolet.  Now healthy flowers are a sign of fertile land, likely to bear fruits and other treasures, and probably prey animals as well; so is it any wonder that humans evolved to be attracted to flowers?  But for there to be some great light written upon the very stars - those huge unsentient balls of burning hydrogen - which also said that flowers were beautiful, now that would be far too much coincidence."

So you explain away the beauty of a flower?

"No, I explain it.  Of course there's a story behind the beauty of flowers and the fact that we find them beautiful.  Behind ordered events, one finds ordered stories; and what has no story is the product of random noise, which is hardly any better.  If you cannot take joy in things that have stories behind them, your life will be empty indeed.  I don't think I take any less joy in a flower than you do; more so, perhaps, because I take joy in its story as well."

Perhaps as you say, there is no surprise from a causal viewpoint - no disruption of the physical order of the universe.  But it still seems to me that, in this creation of humans by evolution, something happened that is precious and marvelous and wonderful.  If we cannot call it a physical miracle, then call it a moral miracle.

"Because it's only a miracle from the perspective of the morality that was produced, thus explaining away all of the apparent coincidence from a merely causal and physical perspective?"

Well... I suppose you could interpret the term that way, yes.  I just meant something that was immensely surprising and wonderful on a moral level, even if it is not surprising on a physical level.

"I think that's what I said."

But it still seems to me that you, from your own view, drain something of that wonder away.

"Then you have problems taking joy in the merely real.  Love has to begin somehow, it has to enter the universe somewhere.  It is like asking how life itself begins - and though you were born of your father and mother, and they arose from their living parents in turn, if you go far and far and far away back, you will finally come to a replicator that arose by pure accident - the border between life and unlife.  So too with love.

"A complex pattern must be explained by a cause which is not already that complex pattern.  Not just the event must be explained, but the very shape and form.  For love to first enter Time, it must come of something that is not love; if this were not possible, then love could not be.

"Even as life itself required that first replicator to come about by accident, parentless but still caused: far, far back in the causal chain that led to you: 3.85 billion years ago, in some little tidal pool.

"Perhaps your children's children will ask how it is that they are capable of love.

"And their parents will say:  Because we, who also love, created you to love.

"And your children's children will ask:  But how is it that you love?

"And their parents will reply:  Because our own parents, who also loved, created us to love in turn.

"Then your children's children will ask:  But where did it all begin?  Where does the recursion end?

"And their parents will say:  Once upon a time, long ago and far away, ever so long ago, there were intelligent beings who were not themselves intelligently designed.  Once upon a time, there were lovers created by something that did not love.

"Once upon a time, when all of civilization was a single galaxy and a single star: and a single planet, a place called Earth.

"Long ago, and far away, ever so long ago."

At at the city, state, or national levels we sometimes "help the poor" via initiatives to "develop" this or that region, but what we mostly have is "welfare" benefits that go directly to individuals.  After all, development funds have a poor track record, and are often diverted by corrupt officials, while direct benefits are the prototype of charity to help those less fortunate than ourselves.  At the international level, however, we mostly have "development aid", even though its track record is just as bad there.  Why don't we give more benefits directly to the world's poor? 

We do not need a strong world government to have a world welfare state - we just need those who want to help the poor to form to a common fund with a common system for distributing benefits.  Some minimal requirements, such as a world ID card, would be made on nations who wanted to let their citizens to get world welfare.  And a special level of international disgust could be reserved for nations that refused to let locals to get world welfare.  Yes this wouldn't be easy, but why does no one even try? 

With a world welfare system donors would have to more directly face their choice between welfare at home and abroad.  Why don't we first ensure everyone in the world at least gets a dollar a day, before we make sure locals don't suffer with only basic cable channels? 

Followup to:  Is Morality Preference?

In the dialogue "Is Morality Preference?", Obert argues for the existence of moral progress by pointing to free speech, democracy, mass street protests against wars, the end of slavery... and we could also cite female suffrage, or the fact that burning a cat alive was once a popular entertainment... and many other things that our ancestors believed were right, but which we have come to see as wrong, or vice versa.

But Subhan points out that if your only measure of progress is to take a difference against your current state, then you can follow a random walk, and still see the appearance of inevitable progress.

One way of refuting the simplest version of this argument, would be to say that we don't automatically think ourselves the very apex of possible morality; that we can imagine our descendants being more moral than us.

But can you concretely imagine a being morally wiser than yourself - one who knows that some particular thing is wrong, when you believe it to be right?

Certainly:  I am not sure of the moral status of chimpanzees, and hence I find it easy to imagine that a future civilization will label them definitely people, and castigate us for failing to cryopreserve the chimpanzees who died in human custody.

Yet this still doesn't prove the existence of moral progress.  Maybe I am simply mistaken about the nature of changes in morality that have previously occurred - like looking at a time chart of "differences between past and present", noting that the difference has been steadily decreasing, and saying, without being able to visualize it, "Extrapolating this chart into the future, we find that the future will be even less different from the present than the present."

So let me throw the question open to my readers:  Whither moral progress?

You might say, perhaps, "Over time, people have become more willing to help one another - that is the very substance and definition of moral progress."

But as John McCarthy put it:

"If everyone were to live for others all the time, life would be like a procession of ants following each other around in a circle."

Once you make "People helping each other more" the definition of moral progress, then people helping each other all the time, is by definition the apex of moral progress.

At the very least we have Moore's Open Question:  It is not clear that helping others all the time is automatically moral progress, whether or not you argue that it is; and so we apparently have some notion of what constitutes "moral progress" that goes beyond the direct identification with "helping others more often".

Or if you identify moral progress with "Democracy!", then at some point there was a first democratic civilization - at some point, people went from having no notion of democracy as a good thing, to inventing the idea of democracy as a good thing.  If increasing democracy is the very substance of moral progress, then how did this moral progress come about to exist in the world?  How did people invent, without knowing it, this very substance of moral progress?

It's easy to come up with concrete examples of moral progress.  Just point to a moral disagreement between past and present civilizations; or point to a disagreement between yourself and present civilization, and claim that future civilizations might agree with you.

It's harder to answer Subhan's challenge - to show directionality, rather than a random walk, on the meta-level.  And explain how this directionality is implemented, on the meta-level: how people go from not having a moral ideal, to having it.

(I have my own ideas about this, as some of you know.  And I'll thank you not to link to them in the comments, or quote them and attribute them to me, until at least 24 hours have passed from this post.)

Greetings, fearless readers:

Due to the Oxford conference on Global Catastrophic Risk, I may miss some posts - possibly quite a few.

Or possibly not.

Just so you don't think I'm dead.

Sincerely,
Eliezer.

University of Pennsylvania professor Diana Mutz's book "Hearing the Other Side: Deliberative versus Participatory Democracy looked at survey evidence as to how often people had conversations with others of differing political viewpoints.  In her words:

One logical conjecture would be to expect this form of political behavior to be much like any other.  In other words, it would be disproportionately the province of well-educated, high-income populations.  Indeed, the frequency of general political discussion tracks closely with these characteristics of high socioeconomic status.  But the correlates of cross-cutting conversation are strikingly different.  As shown in Figure 2.3, there are clear patterns of difference with respect to race, income, and education, but they are not in the usual directions.  Nonwhites are significantly more likely to engage in cross-cutting political conversation than whites.  And as income increases, the frequency of disagreeable conversations declines.   Exposure to disagreement is highest among those who have completed less than a high school degree and lowest among those who have attended graduate school.

As sociologist William Weston notes in discussing Mutz's findings:

I can testify to how easy it is for conversation among academics, the most educated group of people, to turn into a one-position echo chamber. Liberalism is taken to be an IQ test, and the rare conservative is encouraged to be quiet or go elsewhere. For political disagreement I go to the coffee house, which in our town draws a broader range of people than the faculty club contains.

Of course, one explanation would be that what looks like herd behavior and social conformity is really just what happens when a bunch of superior intellects independently settle on the objectively correct viewpoint.  But that's rather a self-serving explanation, isn't it?

One of my pet topics, on which I will post more one of these days, is the Rationalist in Fiction.  Most of the time - it goes almost without saying - the Rationalist is done completely wrong.  In Hollywood, the Rationalist is a villain, or a cold emotionless foil, or a child who has to grow into a real human being, or a fool whose probabilities are all wrong, etcetera.  Even in science fiction, the Rationalist character is rarely done right - bearing the same resemblance to a real rationalist, as the mad scientist genius inventor who designs a new nuclear reactor in a month, bears to real scientists and engineers.

Perhaps this is because most speculative fiction, generally speaking, is interested in someone battling monsters or falling in love or becoming a vampire, or whatever, not in being rational... and it would probably be worse fiction, if the author tried to make that the whole story.  But that can't be the entire problem.  I've read at least one author whose plots are not about rationality, but whose characters are nonetheless, in passing, realistically rational.

That author is Lawrence Watt-Evans.  His work stands out for a number of reasons, the first being that it is genuinely unpredictable.  Not because of a postmodernist contempt for coherence, but because there are events going on outside the hero's story, just like real life.

Most authors, if they set up a fantasy world with a horrible evil villain, and give their main character the one sword that can kill that villain, you could guess that, at the end of the book, the main character is going to kill the evil villain with the sword.

Not Lawrence Watt-Evans.  In a Watt-Evans book, it's entirely possible that the evil villain will die of a heart attack halfway through the book, then the character will decide to sell the sword because they'd rather have the money, and then the character uses the money to set up an investment banking company.

That didn't actually happen in any particular Watt-Evans book - I don't believe in spoilers - but it gives you something of the flavor.

And Watt-Evans doesn't always do this, either - just as, even in real life, things sometimes do go as you expect.

It's this strange realism that charms me about Watt-Evans's work.  It's not done as a schtick, but as faithfulness-to-reality.  Real life doesn't run on perfect rails of dramatic necessity; neither is it random postmodern chaos.  I admire an author who can be faithful to that, and still tell a story.

Watt-Evans's characters, if they happen to be rationalists, are realistic rationalists - they think the same things that you or I would, in their situations.

If the character gets catapulted into a fantasy world, they actually notice the resemblance to their fantasy books, wonder about it, and think to themselves, "If this were a fantasy book, the next thing that would happen is X..." (which may or may not happen, because Watt-Evans doesn't write typical fantasy books).  It's not done as a postmodern self-referential schtick, but as a faithfulness-to-reality; they think what a real rational person would think, in their shoes.

If the character finds out that it is their destiny to destroy the world, they don't waste time on immense dramatic displays - after they get over the shock, they land on their feet and start thinking about it in more or less the fashion that you or I would in their shoes.  Not just, "How do I avoid this?  Are there any possibilities I've overlooked?" but also "Am I sure this is really what's going on?  How reliable is this information?"

If a Watt-Evans character gets their hands on a powerful cheat, they are going to exploit it to the fullest and actively think about creative new ways to use it.  If they find a staff of healing, they're going to set up a hospital.  If they invent a teleportation spell, they're going to think about new industrial uses.

I hate it when some artifact of world-cracking power is introduced and then used as a one-time plot device.  Eventually you get numb, though.

But if a Watt-Evans character finds a device with some interesting magical power halfway through the book, and there's some clever way that the magical power can be used to take over the world, and that character happens to want to take over the world, she's going to say the hell with whatever she was previously doing and go for it.

Most fictional characters are stupid, because they have to be.  This occurs for several reasons; but in speculative fiction, a primary reason is that the author wants to throw around wish-fulfillment superpowers, and the author isn't competent enough to depict the real consequences of halfway intelligent people using that power.

Lawrence Watt-Evans's stories aren't about the intelligence or rationality of his characters.  Nonetheless, Watt-Evans writes intelligent characters, and he's willing to deal with the consequences, which are huge.

Maybe that's the main reason we don't see many realistic rationalists in fiction.

I'd like to see more rationalist fiction.  Not necessarily in Watt-Evans's exact vein, because we already have Watt-Evans, and thus, there is no need to invent him.  But rationalist fiction is hard to do well; there are plenty of cliches out there, but few depictions that say something new or true.

Lawrence Watt-Evans is not going to be everyone's cuppa tea, but if you like SF&F already, give it a try.  Suggested starting books:  The Unwilling Warlord (fantasy), Denner's Wreck (SF).

A stunning hypothesis from the latest Journal of Personality and Social Psychology:

High levels of support often observed for governmental and religious systems can be explained, in part, as a means of coping with the threat posed by chronically or situationally fluctuating levels of perceived personal control. Three experiments demonstrated a causal relation between lowered perceptions of personal control and ... increased beliefs in the existence of a controlling God and defense of the overarching socio-political system.  A 4th experiment showed ... a challenge to the usefulness of external systems of control led to increased illusory perceptions of personal control. ... A cross-national data set demonstrated that lower levels of personal control are associated with higher support for governmental control.

It seems we hope a stronger and more benevolent God or State will protect us when feel less able to protect ourselves.  I'd guess similar effects hold for medicine and media - we believe in doc effectiveness more when we fear out of control of our health, and we believe in media accuracy more when we rely more on their info to protect us.  Can we find data on which beliefs tend to be more biased: confidence in authorities when we feel out of control, or less confidence in authorities when we feel more in control?

Followup to:  Probability is in the Mind

"Reality is that which, when you stop believing in it, doesn't go away."
        -- Philip K. Dick

There are two kinds of Bayesians, allegedly.  Subjective Bayesians believe that "probabilities" are degrees of uncertainty existing in our minds; if you are uncertain about a phenomenon, that is a fact about your state of mind, not a property of the phenomenon itself; probability theory constrains the logical coherence of uncertain beliefs.  Then there are objective Bayesians, who... I'm not quite sure what it means to be an "objective Bayesian"; there are multiple definitions out there.  As best I can tell, an "objective Bayesian" is anyone who uses Bayesian methods and isn't a subjective Bayesian.

If I recall correctly, E. T. Jaynes, master of the art, once described himself as a subjective-objective Bayesian.  Jaynes certainly believed very firmly that probability was in the mind; Jaynes was the one who coined the term Mind Projection Fallacy.  But Jaynes also didn't think that this implied a license to make up whatever priors you liked.  There was only one correct prior distribution to use, given your state of partial information at the start of the problem.

How can something be in the mind, yet still be objective?

It appears to me that a good deal of philosophical maturity consists in being able to keep separate track of nearby concepts, without mixing them up.

For example, to understand evolutionary psychology, you have to keep separate track of the psychological purpose of an act, and the evolutionary pseudo-purposes of the adaptations that execute as the psychology; this is a common failure of newcomers to evolutionary psychology, who read, misunderstand, and thereafter say, "You think you love your children, but you're just trying to maximize your fitness!"

What is it, exactly, that the terms "subjective" and "objective", mean?  Let's say that I hand you a sock.  Is it a subjective or an objective sock?  You believe that 2 + 3 = 5.  Is your belief subjective or objective?  What about two plus three actually equaling five - is that subjective or objective?  What about a specific act of adding two apples and three apples and getting five apples?

I don't intend to confuse you in shrouds of words; but I do mean to point out that, while you may feel that you know very well what is "subjective" or "objective", you might find that you have a bit of trouble saying out loud what those words mean.

Suppose there's a calculator that computes "2 + 3 = 5".  We punch in "2", then "+", then "3", and lo and behold, we see "5" flash on the screen.  We accept this as evidence that 2 + 3 = 5, but we wouldn't say that the calculator's physical output defines the answer to the question 2 + 3 = ?.  A cosmic ray could strike a transistor, which might give us misleading evidence and cause us to believe that 2 + 3 = 6, but it wouldn't affect the actual sum of 2 + 3.

Which proposition is common-sensically true, but philosophically interesting: while we can easily point to the physical location of a symbol on a calculator screen, or observe the result of putting two apples on a table followed by another three apples, it is rather harder to track down the whereabouts of 2 + 3 = 5.  (Did you look in the garage?)

But let us leave aside the question of where the fact 2 + 3 = 5 is located - in the universe, or somewhere else - and consider the assertion that the proposition is "objective".  If a cosmic ray strikes a calculator and makes it output "6" in response to the query "2 + 3 = ?", and you add two apples to a table followed by three apples, then you'll still see five apples on the table.  If you do the calculation in your own head, expending the necessary computing power - we assume that 2 + 3 is a very difficult sum to compute, so that the answer is not immediately obvious to you - then you'll get the answer "5".  So the cosmic ray strike didn't change anything.

And similarly - exactly similarly - what if a cosmic ray strikes a neuron inside your brain, causing you to compute "2 + 3 = 7"?  Then, adding two apples to three apples, you will expect to see seven apples, but instead you will be surprised to see five apples.

If instead we found that no one was ever mistaken about addition problems, and that, moreover, you could change the answer by an act of will, then we might be tempted to call addition "subjective" rather than "objective".  I am not saying that this is everything people mean by "subjective" and "objective", just pointing to one aspect of the concept.  One might summarize this aspect thus:  "If you can change something by thinking differently, it's subjective; if you can't change it by anything you do strictly inside your head, it's objective."

Mind is not magic.  Every act of reasoning that we human beings carry out, is computed within some particular human brain.  But not every computation is about the state of a human brain.  Not every thought that you think is about something that can be changed by thinking.  Herein lies the opportunity for confusion-of-levels.  The quotation is not the referent.  If you are going to consider thoughts as referential at all - if not, I'd like you to explain the mysterious correlation between my thought "2 + 3 = 5" and the observed behavior of apples on tables - then, while the quoted thoughts will always change with thoughts, the referents may or may not be entities that change with changing human thoughts.

The calculator computes "What is 2 + 3?", not "What does this calculator compute as the result of 2 + 3?"  The answer to the former question is 5, but if the calculator were to ask the latter question instead, the result could self-consistently be anything at all!  If the calculator returned 42, then indeed, "What does this calculator compute as the result of 2 + 3?" would in fact be 42.

So just because a computation takes place inside your brain, does not mean that the computation explicitly mentions your brain, that it has your brain as a referent, any more than the calculator mentions the calculator.  The calculator does not attempt to contain a representation of itself, only of numbers.

Indeed, in the most straightforward implementation, the calculator that asks "What does this calculator compute as the answer to the query 2 + 3 = ?" will never return a result, just simulate itself simulating itself until it runs out of memory.

But if you punch the keys "2", "+", and "3", and the calculator proceeds to compute "What do I output when someone punches '2 + 3'?", the resulting computation does have one interesting characteristic: the referent of the computation is highly subjective, since it depends on the computation, and can be made to be anything just by changing the computation.

Is probability, then, subjective or objective?

Well, probability is computed within human brains or other calculators.  A probability is a state of partial information that is possessed by you; if you flip a coin and press it to your arm, the coin is showing heads or tails, but you assign the probability 1/2 until you reveal it.  A friend, who got a tiny but not fully informative peak, might assign a probability of 0.6.

So can you make the probability of winning the lottery be anything you like?

Forget about many-worlds for the moment - you should almost always be able to forget about many-worlds - and pretend that you're living in a single Small World where the lottery has only a single outcome.  You will nonetheless have a need to call upon probability.  Or if you prefer, we can discuss the ten trillionth decimal digit of pi, which I believe is not yet known.  (If you are foolish enough to refuse to assign a probability distribution to this entity, you might pass up an excellent bet, like betting $1 to win $1000 that the digit is not 4.)  Your uncertainty is a state of your mind, of partial information that you possess.  Someone else might have different information, complete or partial.  And the entity itself will only ever take on a single value.

So can you make the probability of winning the lottery, or the probability of the ten trillionth decimal digit of pi equaling 4, be anything you like?

You might be tempted to reply:  "Well, since I currently think the probability of winning the lottery is one in a hundred million, then obviously, I will currently expect that assigning any other probability than this to the lottery, will decrease my expected log-score - or if you prefer a decision-theoretic formulation, I will expect this modification to myself to decrease expected utility.  So, obviously, I will not choose to modify my probability distribution.  It wouldn't be reflectively coherent."

So reflective coherency is the goal, is it?  Too bad you weren't born with a prior that assigned probability 0.9 to winning the lottery!  Then, by exactly the same line of argument, you wouldn't want to assign any probability except 0.9 to winning the lottery.  And you would still be reflectively coherent.  And you would have a 90% probability of winning millions of dollars!  Hooray!

"No, then I would think I had a 90% probability of winning the lottery, but actually, the probability would only be one in a hundred million."

Well, of course you would be expected to say that.  And if you'd been born with a prior that assigned 90% probability to your winning the lottery, you'd consider an alleged probability of 10^-8, and say, "No, then I would think I had almost no probability of winning the lottery, but actually, the probability would be 0.9."

"Yeah?  Then just modify your probability distribution, and buy a lottery ticket, and then wait and see what happens."

What happens?  Either the ticket will win, or it won't.  That's what will happen.  We won't get to see that some particular probability was, in fact, the exactly right probability to assign.

"Perform the experiment a hundred times, and -"

Okay, let's talk about the ten trillionth digit of pi, then.  Single-shot problem, no "long run" you can measure.

Probability is subjectively objective:  Probability exists in your mind: if you're ignorant of a phenomenon, that's an attribute of you, not an attribute of the phenomenon.  Yet it will seem to you that you can't change probabilities by wishing.

You could make yourself compute something else, perhaps, rather than probability.  You could compute "What do I say is the probability?" (answer: anything you say) or "What do I wish were the probability?" (answer: whatever you wish) but these things are not the probability, which is subjectively objective.

The thing about subjectively objective quantities is that they really do seem objective to you.  You don't look them over and say, "Oh, well, of course I don't want to modify my own probability estimate, because no one can just modify their probability estimate; but if I'd been born with a different prior I'd be saying something different, and I wouldn't want to modify that either; and so none of us is superior to anyone else."  That's the way a subjectively subjective quantity would seem.

No, it will seem to you that, if the lottery sells a hundred million tickets, and you don't get a peek at the results, then the probability of a ticket winning, is one in a hundred million.  And that you could be born with different priors but that wouldn't give you any better odds.  And if there's someone next to you saying the same thing about their 90% probability estimate, you'll just shrug and say, "Good luck with that."  You won't expect them to win.

Probability is subjectively really objective, not just subjectively sort of objective.

Jaynes used to recommend that no one ever write out an unconditional probability:  That you never, ever write simply P(A), but always write P(A|I), where I is your prior information.  I'll use Q instead of I, for ease of reading, but Jaynes used I.  Similarly, one would not write P(A|B) for the posterior probability of A given that we learn B, but rather P(A|B,Q), the probability of A given that we learn B and had background information Q.

This is good advice in a purely pragmatic sense, when you see how many false "paradoxes" are generated by accidentally using different prior information in different places.

But it also makes a deep philosophical point as well, which I never saw Jaynes spell out explicitly, but I think he would have approved: there is no such thing as a probability that isn't in any mind.  Any mind that takes in evidence and outputs probability estimates of the next event, remember, can be viewed as a prior - so there is no probability without priors/minds.

You can't unwind the Q.  You can't ask "What is the unconditional probability of our background information being true, P(Q)?"  To make that estimate, you would still need some kind of prior.  No way to unwind back to an ideal ghost of perfect emptiness...

You might argue that you and the lottery-ticket buyer do not really have a disagreement about probability.  You say that the probability of the ticket winning the lottery is one in a hundred million given your prior, P(W|Q1) = 10^-8.  The other fellow says the probability of the ticket winning given his prior is P(W|Q2) = 0.9.  Every time you say "The probability of X is Y", you really mean, "P(X|Q1) = Y".  And when he says, "No, the probability of X is Z", he really means, "P(X|Q2) = Z".

Now you might, if you traced out his mathematical calculations, agree that, indeed, the conditional probability of the ticket winning, given his weird prior is 0.9.  But you wouldn't agree that "the probability of the ticket winning" is 0.9.  Just as he wouldn't agree that "the probability of the ticket winning" is 10^-8.

Even if the two of you refer to different mathematical calculations when you say the word "probability", you don't think that puts you on equal ground, neither of you being better than the other.  And neither does he, of course.

So you see that, subjectively, probability really does feel objective - even after you have subjectively taken all apparent subjectivity into account.

And this is not mistaken, because, by golly, the probability of winning the lottery really is 10^-8, not 0.9.  It's not as if you're doing your probability calculation wrong, after all.  If you weren't worried about being fair or about justifying yourself to philosophers, if you only wanted to get the correct answer, your betting odds would be 10^-8.

Somewhere out in mind design space, there's a mind with any possible prior; but that doesn't mean that you'll say, "All priors are created equal."

When you judge those alternate minds, you'll do so using your own mind - your own beliefs about the universe - your own posterior that came out of your own prior, your own posterior probability assignments P(X|A,B,C,...,Q1).  But there's nothing wrong with that.  It's not like you could judge using something other than yourself.  It's not like you could have a probability assignment without any prior, a degree of uncertainty that isn't in any mind.

And so, when all that is said and done, it still seems like the probability of winning the lottery really  is 10^-8, not 0.9.  No matter what other minds in design space say differently.

Which shouldn't be surprising.  When you compute probabilities, you're thinking about lottery balls, not thinking about brains or mind designs or other people with different priors.  Your probability computation makes no mention of that, any more than it explicitly represents itself.  Your goal, after all, is to win, not to be fair.  So of course probability will seem to be independent of what other minds might think of it.

Okay, but... you still can't win the lottery by assigning a higher probability to winning.

If you like, we could regard probability as an idealized computation, just like 2 + 2 = 4 seems to be independent of any particular error-prone calculator that computes it; and you could regard your mind as trying to approximate this ideal computation.  In which case, it is good that your mind does not mention people's opinions, and only thinks of the lottery balls; the ideal computation makes no mention of people's opinions, and we are trying to reflect this ideal as accurately as possible...

But what you will calculate is the "ideal calculation" to plug into your betting odds, will depend on your prior, even though the calculation won't have an explicit dependency on "your prior".  Someone who thought the universe was anti-Occamian, would advocate an anti-Occamian calculation, regardless of whether or not anyone thought the universe was anti-Occamian.

Your calculations get checked against reality, in a probabilistic way; you either win the lottery or not.  But interpreting these results, is done with your prior; once again there is no probability that isn't in any mind.

I am not trying to argue that you can win the lottery by wishing, of course.  Rather, I am trying to inculcate the ability to distinguish between levels.

When you think about the ontological nature of probability, and perform reductionism on it - when you try to explain how "probability" fits into a universe in which states of mind do not exist fundamentally - then you find that probability is computed within a brain; and you find that other possible minds could perform mostly-analogous operations with different priors and arrive at different answers.

But, when you consider probability as probability, think about the referent instead of the thought process - which thinking you will do in your own thoughts, which are physical processes - then you will conclude that the vast majority of possible priors are probably wrong.  (You will also be able to conceive of priors which are, in fact, better than yours, because they assign more probability to the actual outcome; you just won't know in advance which alternative prior is the truly better one.)

If you again swap your goggles to think about how probability is implemented in the brain, the seeming objectivity of probability is the way the probability algorithm feels from inside; so it's no mystery that, considering probability as probability, you feel that it's not subject to your whims.  That's just what the probability-computation would be expected to say, since the computation doesn't represent any dependency on your whims.

But when you swap out those goggles and go back to thinking about probabilities, then, by golly, your algorithm seems to be right in computing that probability is not subject to your whims.  You can't win the lottery just by changing your beliefs about it.  And if that is the way you would be expected to feel, then so what?  The feeling has been explained, not explained away; it is not a mere feeling.  Just because a calculation is implemented in your brain, doesn't mean it's wrong, after all.

Your "probability that the ten trillionth decimal digit of pi is 4", is an attribute of yourself, and exists in your mind; the real digit is either 4 or not.  And if you could change your belief about the probability by editing your brain, you wouldn't expect that to change the probability.

Therefore I say of probability that it is "subjectively objective".

Followup to:  Fundamental Doubts, Where Recursive Justification Hits Bottom, No Universally Compelling Arguments, Joy in the Merely Real, Evolutionary Psychology

"Let us understand, once and for all, that the ethical progress of society depends, not on imitating the cosmic process, still less in running away from it, but in combating it."
        -- T. H. Huxley ("Darwin's bulldog", early advocate of evolutionary theory)

There is a quote from some Zen Master or other, who said something along the lines of:

"Western man believes that he is rebelling against nature, but he does not realize that, in doing so, he is acting according to nature."

The Reductionist Masters of the West, strong in their own Art, are not so foolish; they do realize that they always act within Nature.

You can narrow your focus and rebel against a facet of existing Nature - polio, say - but in so doing, you act within the whole of Nature.  The syringe that carries the polio vaccine is forged of atoms; our minds, that understood the method, embodied in neurons.  If Jonas Salk had to fight laziness, he fought something that evolution instilled in him - a reluctance to work that conserves energy.  And he fought it with other emotions that natural selection also inscribed in him: feelings of friendship that he extended to humanity, heroism to protect his tribe, maybe an explicit desire for fame that he never acknowledged to himself - who knows?  (I haven't actually read a biography of Salk.)

The point is, you can't fight Nature from beyond Nature, only from within it.  There is no acausal fulcrum on which to stand outside reality and move it.  There is no ghost of perfect emptiness by which you can judge your brain from outside your brain.  You can fight the cosmic process, but only by recruiting other abilities that evolution originally gave to you.

And if you fight one emotion within yourself - looking upon your own nature, and judging yourself less than you think should be - saying perhaps, "I should not want to kill my enemies" - then you make that judgment, by...

How exactly does one go about rebelling against one's own goal system?

From within it, naturally.

This is perhaps the primary thing that I didn't quite understand as a teenager.

At the age of fifteen (fourteen?), I picked up a copy of TIME magazine and read an article on evolutionary psychology.  It seemed like one of the most massively obvious-in-retrospect ideas I'd ever heard.  I went on to read The Moral Animal by Robert Wright.  And later The Adapted Mind - but from the perspective of personal epiphanies, The Moral Animal pretty much did the job.

I'm reasonably sure that if I had not known the basics of evolutionary psychology from my teenage years, I would not currently exist as the Eliezer Yudkowsky you know.

Indeed, let me drop back a bit further:

At the age of... I think it was nine... I discovered the truth about sex by looking it up in my parents' home copy of the Encyclopedia Britannica (stop that laughing).  Shortly after, I learned a good deal more by discovering where my parents had hidden the secret 15th volume of my long-beloved Childcraft series.  I'd been avidly reading the first 14 volumes - some of them, anyway - since the age of five.  But the 15th volume wasn't meant for me- it was the "Guide for Parents".

The 15th volume of Childcraft described the life cycle of children.  It described the horrible confusion of the teenage years - teenagers experimenting with alcohol, with drugs, with unsafe sex, with reckless driving, the hormones taking over their minds, the overwhelming importance of peer pressure, the tearful accusations of "You don't love me!" and "I hate you!"

I took one look at that description, at the tender age of nine, and said to myself in quiet revulsion, I'm not going to do that.

And I didn't.

My teenage years were not untroubled.  But I didn't do any of the things that the Guide to Parents warned me against.  I didn't drink, drive, drug, lose control to hormones, pay any attention to peer pressure, or ever once think that my parents didn't love me.

In a safer world, I would have wished for my parents to have hidden that book better.

But in this world, which needs me as I am, I don't regret finding it.

I still rebelled, of course.  I rebelled against the rebellious nature the Guide to Parents described to me.  That was part of how I defined my identity in my teenage years - "I'm not doing the standard stupid stuff."  Some of the time, this just meant that I invented amazing new stupidity, but in fact that was a major improvement.

Years later, The Moral Animal made suddenly obvious the why of all that disastrous behavior I'd been warned against.  Not that Robert Wright pointed any of this out explicitly, but it was obvious given the elementary concept of evolutionary psychology:

Physiologically adult humans are not meant to spend an additional 10 years in a school system; their brains map that onto "I have been assigned low tribal status".  And so, of course, they plot rebellion - accuse the existing tribal overlords of corruption - plot perhaps to split off their own little tribe in the savanna, not realizing that this is impossible in the Modern World.  The teenage males map their own fathers onto the role of "tribal chief"...

Echoes in time, thousands of repeated generations in the savanna carving the pattern, ancient repetitions of form, reproduced in the present in strange twisted mappings, across genes that didn't know anything had changed...

The world grew older, of a sudden.

And I'm not going to go into the evolutionary psychology of "teenagers" in detail, not now, because that would deserve its own post.

But when I read The Moral Animal, the world suddenly acquired causal depth.  Human emotions existed for reasons, they weren't just unexamined givens.  I might previously have questioned whether an emotion was appropriate to its circumstance - whether it made sense to hate your parents, if they did really love you - but I wouldn't have thought, before then, to judge the existence of hatred as an evolved emotion.

And then, having come so far, and having avoided with instinctive ease all the classic errors that evolutionary psychologists are traditionally warned against - I was never once tempted to confuse evolutionary causation with psychological causation - I went wrong at the last turn.

The echo in time that was teenage psychology was obviously wrong and stupid - a distortion in the way things should be - so clearly you were supposed to unwind past it, compensate in the opposite direction or disable the feeling, to arrive at the correct answer.

It's hard for me to remember exactly what I was thinking in this era, but I think I tended to focus on one facet of human psychology at any given moment, trying to unwind myself a piece at a time.  IIRC I did think, in full generality, "Evolution is bad; the effect it has on psychology is bad."  (Like it had some kind of "effect" that could be isolated!)  But somehow, I managed not to get to "Evolutionary psychology is the cause of altruism; altruism is bad."

It was easy for me to see all sorts of warped altruism as having been warped by evolution.

People who wanted to trust themselves with power, for the good of their tribe - that had an obvious evolutionary explanation; it was, therefore, a distortion to be corrected.

People who wanted to be altruistic in ways their friends would approve of - obvious evolutionary explanation; therefore a distortion to be corrected.

People who wanted to be altruistic in a way that would optimize their fame and repute - obvious evolutionary distortion to be corrected.

People who wanted to help only their family, or only their nation - acting out ancient selection pressures on the savanna; move past it.

But the fundamental will to help people?

Well, the notion of that being merely evolved, was something that, somehow, I managed to never quite accept.  Even though, in retrospect, the causality is just as obvious as teen revolutionism.

IIRC, I did think something along the lines of:  "Once you unwind past evolution, then the true morality isn't likely to contain a clause saying, 'This person matters but this person doesn't', so everyone should matter equally, so you should be as eager to help others as help yourself."  And so I thought that even if the emotion of altruism had merely evolved, it was a right emotion, and I should keep it.

But why think that people mattered at all, if you were trying to unwind past all evolutionary psychology?  Why think that it was better for people to be happy than sad, rather than the converse?

If I recall correctly, I did ask myself that, and sort of waved my hands mentally and said, "It just seems like one of the best guesses - I mean, I don't know that people are valuable, but I can't think of what else could be."

This is the Avoiding Your Belief's Real Weak Points / Not Spontaneously Thinking About Your Belief's Most Painful Weaknesses antipattern in full glory:  Get just far enough to place yourself on the first fringes of real distress, and then stop thinking.

And also the antipattern of trying to unwind past everything that is causally responsible for your existence as a mind, to arrive at a perfectly reliable ghost of perfect emptiness.

Later, having also seen others making similar mistakes, it seems to me that the general problem is an illusion of mind-independence that comes from picking something that appeals to you, while still seeming philosophically simple.

As if the appeal to you, of the moral argument, weren't still a feature of your particular point in mind design space.

As if there weren't still an ordinary and explicable causal history behind the appeal, and your selection of that particular principle.

As if, by making things philosophically simpler-seeming, you could enhance their appeal to a ghost-in-the-machine who would hear your justifications starting from scratch, as fairness demands.

As if your very sense of simplicity were not an aesthetic sense inscribed in you by evolution.

As if your very intuitions of "moral argument" and "justification", were not an architecture-of-reasoning inscribed in you by natural selection, and just as causally explicable as any other feature of human psychology...

You can't throw away evolution, and end up with a perfectly moral creature that humans would have been, if only we had never evolved; that's really not how it works.

Why accept intuitively appealing arguments about the nature of morality, rather than intuitively unappealing ones, if you're going to distrust everything in you that ever evolved?

Then what is right?  What should we do, having been inscribed by a blind mad idiot god whose incarnation-into-reality takes the form of millions of years of ancestral murder and war?

But even this question - every fragment of it - the notion that a blind mad idiocy is an ugly property for a god to have, or that murder is a poisoned well of order, even the words "right" and "should" - all a phenomenon within nature.  All traceable back to debates built around arguments appealing to intuitions that evolved in me.

You can't jump out of the system.  You really can't.  Even wanting to jump out of the system - the sense that something isn't justified "just because it evolved" - is something that you feel from within the system.  Anything you might try to use to jump - any sense of what morality should be like, if you could unwind past evolution - is also there as a causal result of evolution.

Not everything we think about morality is directly inscribed by evolution, of course.  We have values that we got from our parents teaching them to us as we grew up; after it won out in a civilizational debate conducted with reference to other moral principles; that were themselves argued into existence by appealing to built-in emotions; using an architecture-of-interpersonal-moral-argument that evolution burped into existence.

It all goes back to evolution.  This doesn't just include things like instinctive concepts of fairness, or empathy, it includes the whole notion of arguing morals as if they were propositional beliefs.  Evolution created within you that frame of reference within which you can formulate the  concept of moral questioning.  Including questioning evolution's fitness to create our moral frame of reference.  If you really try to unwind outside the system, you'll unwind your unwinders.

That's what I didn't quite get, those years ago.

I do plan to dissolve the cognitive confusion that makes words like "right" and "should" seem difficult to grasp.  I've been working up to that for a while now.

But I'm not there yet, and so, for now, I'm going to jump ahead and peek at an answer I'll only later be able to justify as moral philosophy:

Embrace reflection.  You can't unwind to emptiness, but you can bootstrap from a starting point.

Go on morally questioning the existence (and not just appropriateness) of emotions.  But don't treat the mere fact of their having evolved as a reason to reject them.  Yes, I know that "X evolved" doesn't seem like a good justification for having an emotion; but don't let that be a reason to reject X, any more than it's a reason to accept it.  Hence the post on the Genetic Fallacy: causation is conceptually distinct from justification.  If you try to apply the Genetic Accusation to automatically convict and expel your genes, you're going to run into foundational trouble - so don't!

Just ask if the emotion is justified - don't treat its evolutionary cause as proof of mere distortion.  Use your current mind to examine the emotion's pluses and minuses, without being ashamed; use your full strength of morality.

Judge emotions as emotions, not as evolutionary relics.  When you say, "motherly love outcompeted its alternative alleles because it protected children that could carry the allele for motherly love", this is only a cause, not a sum of all moral arguments.  The evolutionary psychology may grant you helpful insight into the pattern and process of motherly love, but it neither justifies the emotion as natural, nor convicts it as coming from an unworthy source.  You don't make the Genetic Accusation either way.  You just, y'know, think about motherly love, and ask yourself if it seems like a good thing or not; considering its effects, not its source.

You tot up the balance of moral justifications, using your current mind - without worrying about the fact that the entire debate takes place within an evolved framework.

That's the moral normality to which my yet-to-be-revealed moral philosophy will add up.

And if, in the meanwhile, it seems to you like I've just proved that there is no morality... well, I haven't proved any such thing.  But, meanwhile, just ask yourself if you might want to help people even if there were no morality.  If you find that the answer is yes, then you will later discover that you discovered morality.

From a new study in Journal of Personality and Social Psychology:

People underestimated by as much as 50% the likelihood that others would agree to a direct request for help, across a range of requests occurring in both experimental and natural field settings. ... Experimentally manipulating a person's perspective (as help seeker or potential helper) could elicit this underestimation effect. ... Help seekers were less willing than potential helpers were to appreciate the social costs of refusing a direct request for help.

We don't like to ask for help, men especially, because asking threatens our status.  Believing that others won't help lets us "sincerely" avoid asking.

Followup to:  The Genetic Fallacy, Where Recursive Justification Hits Bottom

Yesterday I said that - because humans are not perfect Bayesians - the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.

This is one of the most difficult techniques of rationality (on which I will separately post, one of these days).  Descartes, setting out to "doubt, insofar as possible, all things", ended up trying to prove the existence of God - which, if he wasn't a secret atheist trying to avoid getting burned at the stake, is pretty pathetic.  It is hard to doubt an idea to which we are deeply attached; our mind naturally reaches for cached thoughts and rehearsed arguments.

But today's post concerns a different kind of difficulty - the case where the doubt is so deep, of a source so fundamental, that you can't make a true fresh beginning.

Case in point:  Remember when, in the The Matrix, Morpheus told Neo that the machines were harvesting the body heat of humans for energy, and liquefying the dead to feed to babies?  I suppose you thought something like, "Hey!  That violates the second law of thermodynamics."

Well, it does violate the second law of thermodynamics.  But if the Matrix's makers had cared about the flaw once it was pointed out to them, they could have fixed the plot hole in any of the sequels, in fifteen seconds, this easily:

Neo:  "Doesn't harvesting human body heat for energy, violate the laws of thermodynamics?"

Morpheus:  "Where'd you learn about thermodynamics, Neo?"

Neo:  "In school."

Morpheus:  "Where'd you go to school, Neo?"

Neo:  "Oh."

Morpheus:  "The machines tell elegant lies."

Now, mind you, I am not saying that this excuses the original mistake in the script.  When my mind generated this excuse, it came clearly labeled with that warning sign of which I have spoken, "Tada!  Your mind can generate an excuse for anything!"  You do not need to tell me that my plot-hole-patch is a nitwit idea, I am well aware of that...

...but, in point of fact, if you woke up out of a virtual reality pod one day, you would have to suspect all the physics you knew.  Even if you looked down and saw that you had hands, you couldn't rely on there being blood and bone inside them.  Even if you looked up and saw stars, you couldn't rely on their being trillions of miles away.  And even if you found yourself thinking, you couldn't rely on your head containing a brain.

You could still try to doubt, even so.  You could do your best to unwind your thoughts past every lesson in school, every science paper read, every sensory experience, every math proof whose seeming approval by other mathematicians might have been choreographed to conceal a subtle flaw...

But suppose you discovered that you were a computer program and that the Dark Lords of the Matrix were actively tampering with your thoughts.

Well... in that scenario, you're pretty much screwed, I'd have to say.

Descartes vastly underestimated the powers of an infinitely powerful deceiving demon when he supposed he could trust "I think therefore I am."  Maybe that's just what they want you to think.  Maybe they just inserted that conclusion into your mind with a memory of it seeming to have an irrefutable chain of logical support, along with some peer pressure to label it "unquestionable" just like all your friends.

(Personally, I don't trust "I think therefore I am" even in real life, since it contains a term "am" whose meaning I find confusing, and I've learned to spread my confidence intervals very widely in the presence of basic confusion.  As for absolute certainty, don't be silly.)

Every memory of justification could be faked.  Every feeling of support could be artificially induced.  Modus ponens could be a lie.  Your concept of "rational justification" - not just your specific concept, but your notion that any such thing exists at all - could have been manufactured to mislead you.  Your trust in Reason itself could have been inculcated to throw you off the trail.

So you might as well not think about the possibility that you're a brain with choreographed thoughts, because there's nothing you can do about it...

Unless, of course, that's what they want you to think.

Past a certain level of doubt, it's not possible to start over fresh.  There's nothing you can unassume to find some firm rock on which to stand.  You cannot unwind yourself into a perfectly empty and perfectly reliable ghost in the machine.

This level of meta-suspicion should be a rare occasion.  For example, suspecting that all academic science is an organized conspiracy, should not run into anything like these meta-difficulties.  Certainly, someone does not get to plead that unwinding past the Bible is impossible because it is too foundational; atheists walk the Earth without falling into comas.  Remember, when Descartes tried to outwit an infinitely powerful deceiving demon, he first tried to make himself absolutely certain of a highly confusing statement, and then proved the existence of God.  Consider that a caution about what you try to claim is "too basic for a fresh beginning".  And even basic things can still be doubted, it is only that we use our untrustworthy brains to doubt them.

Or consider the case of our existence as evolved brains.  Natural selection isn't trustworthy, and we have specific reason to suspect it.  We know that evolution is stupid.  We know many specific ways in which our human brains fail, taken beyond the savanna.  But you can't clear your mind of evolutionary influences and start over.  It would be like deciding that you don't trust neurons, so you're going to clear your mind of brains.

And evolution certainly gets a chance to influence every single thought that runs through your mind!  It is the very reason why you exist as a thinker, rather than a lump of carbon - and that doesn't mean evolution summoned a ghost-in-the-machine into you; it designed the ghost.  If you learn culture, it is because you were built to learn culture.

But in fact, we don't run into unmanageable meta-trouble in trying to come up with specific patches for specific known evolved biases.  And evolution is stupid, so even though it has set up self-deceptive circuits in us, these circuits are not infinitely difficult to comprehend and outwit.

Or so it seems!  But it really does seem that way, on reflection.

There is no button you can press to rewind past your noisy brain, and become a perfectly reliable ghost of perfect emptiness.  That's not just because your brain is you.  It's also because you can't unassume things like modus ponens or belief updating.  You can unassume them as explicit premises for deliberate reasoning - a hunter-gatherer has no explicit concept of modus ponens - but you can't delete the actual dynamics (and all their products!)

So, in the end, I think we must allow the use of brains to think about thinking; and the use of evolved brains to think about evolution; and the use of inductive brains to think about induction; and the use of brains with an Occam prior to think about whether the universe appears to be simple; for these things we really cannot unwind entirely, even when we have reason to distrust them.  Strange loops through the meta level, I think, are not the same as circular logic.

From a thoughtful essay by William Deresiewicz:

An elite education ... makes you incapable of talking to people who aren't like you. Elite schools pride themselves on their diversity, but that diversity is almost entirely a matter of ethnicity and race. With respect to class, these schools are largely - indeed increasingly - homogeneous.  ... My education taught me to believe that people who didn't go to an Ivy League or equivalent school weren't worth talking to, regardless of their class. ... Elite universities ... select for and develop one form of intelligence: the analytic. ... social intelligence and emotional intelligence and creative ability, to name just three other forms, are not distributed preferentially among the educational elite. ... There are due dates and attendance requirements at places like Yale, but no one takes them very seriously. ... Students ... get an endless string of second chances. Not so at places like Cleveland State.  ..

An elite education gives you the chance to be rich - which is, after all, what we're talking about - but it takes away the chance not to be. ... [If they] pursue a riskier or less lucrative course after graduation ... [elite students] tend to give up more quickly than others. .. a couple of graduate students ... were talking about trying to write poetry, how friends of theirs from college called it quits within a year or two while people they know from less prestigious schools are still at it. ...

The final and most damning disadvantage of an elite education: that it is profoundly anti-intellectual. ... Being an intellectual is not the same as being smart. Being an intellectual means more than doing your homework. ... They are products of a system that rarely asked them to think about something bigger than the next assignment. ... Being an intellectual means, first of all, being passionate about ideas - and not just for the duration of a semester, for the sake of pleasing the teacher, or for getting a good grade. ... Students at Yale and Columbia ... have seemed content to color within the lines that their education had marked out for them. Only a small minority have seen their education as part of a larger intellectual journey, have approached the work of the mind with a pilgrim soul. ...  Places like Yale are simply not set up to help students ask the big questions.

My experience confirms all of this.  The sort of risk-taking, soul-searching, and success-sacrifice that is required for (but hardly guarantees) truly great intellectual achievement is not much rewarded in our current elite education system.

In lists of logical fallacies, you will find included "the genetic fallacy" - the fallacy attacking a belief, based on someone's causes for believing it.

This is, at first sight, a very strange idea - if the causes of a belief do not determine its systematic reliability, what does?  If Deep Blue advises us of a chess move, we trust it based on our understanding of the code that searches the game tree, being unable to evaluate the actual game tree ourselves.  What could license any probability assignment as "rational", except that it was produced by some systematically reliable process?

Articles on the genetic fallacy will tell you that genetic reasoning is not always a fallacy - that the origin of evidence can be relevant to its evaluation, as in the case of a trusted expert.  But other times, say the articles, it is a fallacy; the chemist Kekulé first saw the ring structure of benzene in a dream, but this doesn't mean we can never trust this belief.

So sometimes the genetic fallacy is a fallacy, and sometimes it's not?

The genetic fallacy is formally a fallacy, because the original cause of a belief is not the same as its current justificational status, the sum of all the support and antisupport currently known.

Yet we change our minds less often than we think.  Genetic accusations have a force among humans that they would not have among ideal Bayesians.

Clearing your mind is a powerful heuristic when you're faced with new suspicion that many of your ideas may have come from a flawed source.

Once an idea gets into our heads, it's not always easy for evidence to root it out.  Consider all the people out there who grew up believing in the Bible; later came to reject (on a deliberate level) the idea that the Bible was written by the hand of God; and who nonetheless think that the Bible contains indispensable ethical wisdom.  They have failed to clear their minds; they could do significantly better by doubting anything the Bible said because the Bible said it.

At the same time, they would have to bear firmly in mind the principle that reversed stupidity is not intelligence; the goal is to genuinely shake your mind loose and do independent thinking, not to negate the Bible and let that be your algorithm.

Once an idea gets into your head, you tend to find support for it everywhere you look - and so when the original source is suddenly cast into suspicion, you would be very wise indeed to suspect all the leaves that originally grew on that branch...

If you can!  It's not easy to clear your mind.  It takes a convulsive effort to actually reconsider, instead of letting your mind fall into the pattern of rehearsing cached arguments.  "It ain't a true crisis of faith unless things could just as easily go either way," said Thor Shenkel.

You should be extremely suspicious if you have many ideas suggested by a source that you now know to be untrustworthy, but by golly, it seems that all the ideas still ended up being right - the Bible being the obvious archetypal example.

On the other hand... there's such a thing as sufficiently clear-cut evidence, that it no longer significantly matters where the idea originally came from.  Accumulating that kind of clear-cut evidence is what Science is all about.  It doesn't matter any more that Kekulé first saw the ring structure of benzene in a dream - it wouldn't matter if we'd found the hypothesis to test by generating random computer images, or from a spiritualist revealed as a fraud, or even from the Bible.  The ring structure of benzene is pinned down by enough experimental evidence to make the source of the suggestion irrelevant.

In the absence of such clear-cut evidence, then you do need to pay attention to the original sources of ideas - to give experts more credence than layfolk, if their field has earned respect - to suspect ideas you originally got from suspicious sources - to distrust those whose motives are untrustworthy, if they cannot present arguments independent of their own authority.

The genetic fallacy is a fallacy when there exist justifications beyond the genetic fact asserted, but the genetic accusation is presented as if it settled the issue.

Some good rules of thumb (for humans):

• Be suspicious of genetic accusations against beliefs that you dislike, especially if the proponent claims justifications beyond the simple authority of a speaker.  "Flight is a religious idea, so the Wright Brothers must be liars" is one of the classically given examples.
• By the same token, don't think you can get good information about a technical issue just by sagely psychoanalyzing the personalities involved and their flawed motives.  If technical arguments exist, they get priority.
• When new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves that grew from that root.  You are not licensed to reject them outright as conclusions, because reversed stupidity is not intelligence, but...
• Be extremely suspicious if you find that you still believe the early suggestions of a source you later rejected.

Added:  Hal Finney suggests that we should call it "the genetic heuristic".

Followup to:  Where Recursive Justification Hits Bottom

In "Where Recursive Justification Hits Bottom", I concluded that it's okay to use induction to reason about the probability that induction will work in the future, given that it's worked in the past; or to use Occam's Razor to conclude that the simplest explanation for why Occam's Razor works is that the universe itself is fundamentally simple.

Now I am far from the first person to consider reflective application of reasoning principles.  Chris Hibbert compared my view to Bartley's Pan-Critical Rationalism (I was wondering whether that would happen).  So it seems worthwhile to state what I see as the distinguishing features of my view of r