I like this question, mostly because it is what I am boning up on right now for a lecture at the Ashokan seminar next week.
Work or strain hardening, as has already been said, is the result of plastic deformation due to slip. Slip is s shearing action that occurs across planes in the atomic lattices of metals. This shearing action can only occur along planes of the closest atomic packing.
Example: take a whole bunch of ping pong balls and glue them to 2 boards on a grid of 1" spacing. Now do the same with two boards at 2" spacing. Put the two 2" spaced, together ping-pong balls to ping pong balls and tilt the stack. You will have to go a very steep angle in order for the top board to slide. Repeat with the 1" spaced boards and you will not have to go near as far in your tilting to get "slip".
The tightest ping-pong ball atomic lattice was more susceptible to slip. Now we are going to get techy- when you heat steel to Ac1 (what SOME may call critical temp) the iron atoms will shift from a body centered cubic (bcc) configuration to a face centered cubic (fcc) configuration. bcc is less tightly packed than fcc which is close packed, this is why steel moves so much easier at proper forging temperature.
Now slip will most often occur at certain angles within these configurations, imagine this angle like 45 degrees, 0 degrees won't slip and 90 degrees won't slip.
Example: take a deck of cards, an object consisting of many planes upon which slip can occur and the deck will deform. Lay the cards flat and push straight down and nothing moves, set the deck on edge and push straight down and nothing moves, now set the deck at a 45 degree and push straight down and the deck easily changes shape.
Now the 45-degree thing is represented in tech talk by indices (Miller indices) relating to a 3 dimensional grid and then number that corresponds with the 45 in fcc is (1,1,1). In bcc metals, like iron at room temp, the (1,1,1) plane is not available due to the way the atoms are stacked so it is much harder to get things to move.
Let's complicate things even more, you may regret asking this question

. The theoretical force required for slip is much, much higher than what reality presents us with- does the math lie? No. Here is where the aforementioned "line" and "screw" defects come into play, mostly just the line one though. The actual term for these are "edge dislocations" in the atomic stacking and they are incomplete or half rows in the stack that creates distortions in the lattice. What this does is creates and easier avenue for slip. If you have a huge area rug that you wish to move across a rough floor trying to slide the whole thing will take a lot of force (the theoretical numbers), but if you create a wrinkle in the rug the whole thing can be moved just by pushing that wrinkle down the line, this is how edge dislocations make slip easier. Easier until....
Now we get to the heart of your question. When these edge dislocations meet an obstacle like a grain boundary, a serious lattice defect or a large substitional atom (like a big old chromium in the middle of all those irons) they start to pile up and you get a serious dislocation traffic jam. To add to this, the process of cold working continually creates more dislocations. Now as soon as you heat the steal above Ac1 all will be erased and the whole thing can start over on a clean slate, that is annealing)
Very simple explanation: Imagine cold working like a huge cube in your back yard that is made up of hundreds of smaller blocks. Pick a row of blocks, any row, in that cube and give it a push. They slide and misalign to the rest of the cube. Now keep pushing from different directions on different rows and eventually the half dislocations will start to multiply and interfere with each other. Soon it will get very difficult to find a row that is not tangled with another and if you keep pushing the block may come tumbling down!

This would be fracture in a metal that was pushed too far. If you were to reheat the metal it would be the same as setting all of the blocks in your cube back to the same straight and ordered stack you started with.
With extreme cold working the grains, that are not aligned to a plane that can slip, will deform in a process of deformation known as twinning (which I wont get into here) or until they rotate or align themselves in a direction that slip can occur. This will eventually result in most of the grains being elongated in the direction of working resulting in different properties in that direction than a direction perpendicular to it. That is, it will be less ductile and have higher strength along its length but if you were to cross section it and, check it the other way (perpendicular) you would find the opposite to be true. It cannot be stresses enough that
any heating approaching Ac1 will completely erase all of these effects.