18. Positive Feedback is More Efficient Than Negative Feedback - A Geometric Approach

(Epistemic status: a hastily-written recreation of something I remember reading once and can’t find anymore; if you can find it let me know. The original post had some good images. Probably best read after post 11, “Why the First “High Dimension” is Six or Maybe Five”.)

Positive reinforcement is just plain more efficient and effective than negative reinforcement, if both are feasible, and I can pretty straightforwardly present a model that argues strongly for it. Consider the following setup: we’re trying to get some tiny simple agent to navigate to a goal area within some simple space. No overly complex obstacles, no particular hazards, just a tiny simple agent capable of approaching or avoiding marked areas and a goal with a strong but extremely short-range attractiveness. We have negative feedback markers, which the agent will strive to avoid, and positive feedback markers, which the agent will try to approach; we can model these both as having some infinite-range repulsive or attractive force imparted to the moving agent with some appropriate falloff inversely dependent on the distance. (I’m using valence terminology here, not control-system terminology.)

Let’s say our agent lives in some nice low-dimensional space - a line segment, say, or some region of the plane. Using markers of only one flavor, how many will we need? We can assume, to put a thumb on the scale, that we can ensure that our little agent starts closer to the goal than our markers, if we want. If we use positive feedback markers, we only ever need one of them - just put it right on the goal area to augment the goal’s weak stickiness and be done; the agent will approach it straightforwardly. On the other hand, for the line segment, we need two negative feedback markers to bound the agent, and in the plane region, we’ll likely need at least three or four, depending on just how far from the goal the agent starts.

What if the agent is in some slightly higher-dimensional space, like our familiar three-dimensional space? It’s a little harder to see, but if we’re using negative markers we’ll now need something like six to eight of them at the least, and more likely more than that - but still just one positive marker. Hopefully you can see the pattern here - as the dimension D keeps growing, we’ll need easily 2*D negative markers and generally more like 2^D of them, if not many more - and yet we will only ever need a single positive marker to draw the agent in.

Now consider just how high-dimensional the space of behaviors really is - especially the kinds of complex behaviors you might hope to elicit from your fellow people, or even animals! Successively downvoting undesirable behaviors will take lots of time and effort and exhaust your training subject’s patience and energy fairly quickly; helping them out by telling or showing them what you want and marking and praising that behavior will show returns with vastly better speed and surety. The same is naturally true of self-modifications you wish to enact. Rather than punish yourself for doing the things you don’t want, or ruling out and pruning away behaviors you don’t want, you should preferentially figure out what you do in fact want yourself to do and reward yourself when you do it; this will be vastly more effective.

The moral here is: don’t tell people what you don’t want them to do when it’s vastly easier for everyone involved to just tell them what you want from them. They’ll know what you want, you’ll get what you want, and everyone will be much the happier for it.


21261/32768

Comments

Post a Comment

Popular posts from this blog

4. Seven-ish Words from My Thought-Language

Seven entries from a dictionary that will never exist, even though it should. The words in [brackets] show up with some frequency in my thoughts, and I struggle with English’s semantic poverty. Some of them show up in other languages, like Korean or Lojban. Others among them, people I’ve spoken to share in wishing were wider-used words. [untranslatable 1: word-clogged/interference-silent] (adj): Of a person or their state of mind. Descriptive of the phenomenon where a person has multiple different things to say about a specific topic, or multiple different responses or reactions to a remark an interlocutor has made, only one of which can be taken. To be tongue-tied not out of a lack of things to say, but out of an overabundance of mutually exclusive things to say, all of equal priority, especially when each thing to say would result in following importantly different conversational paths. Often used instead in its natural verb form, which means “to be silent as a result of [word-clog]”...
Read more

0. 31 Posts, 1k Words, 2^15 Total

Image
    The received wisdom - or so friends and trusted advisors and internet randos (and in particular DF, who is all three) have told me - is that if you want to get good at something, you have to go do a lot of it, and in particular, that if you want to get good at writing, expressing yourself, and rapidly clarifying ideas for a relatively public audience, the indicated thing to do is to write a long series of blog posts. Separately, people I speak to on topics close to my heart - mathematics, yes, but also aspects of phenomenology, mechanism design, mental health, personal history, and stranger topics yet - have regularly bemoaned the fact that I lack a central repository of all the delicious shiny drippings from my brainpan, or at least have asked me where such a thing might be found and seemed sad when I firmly informed them that such a thing did not exist and quite likely never would. (To all such people: I would apologize for my lateness, but it is in the nature of a mathc...
Read more

11. Why the First “High Dimension” is Six or Maybe Five

Image
(Epistemic status: morally correct, in the mathematician’s sense; here to give flavor and intuition without too much rigor.) People toss around the concept of “high-dimensional space” a lot without having a good idea of where that starts. Interpretability researchers staring at vectors in hundreds of dimensions are clearly contemplating a high-dimensional space - loss landscapes packed full of saddle points and spheres where any given pair of vectors is almost certainly nearly orthogonal - but do they know what it means to be a high-dimensional space? Surely we can do better for ourselves than Justice Stewart’s infamous maxim - “I know it when I see it”! What do we care about, when we want to tell how many orthogonal axes we need in a space to start calling it high-dimensional? I claim that we should care about looking at unit-radius disks - like, literal filled-in two-dimensional circles - and in particular, checking when, whether, and how pairs of those unit-radius disks overlap when...
Read more