12-28-2015, 02:36 PM
Can't say *too* much about the secret sauce before I have papers at journals, but I can give a quick outline.
The problem was that in conventional neural network architecture all the connections have the *SAME* learning rate. If the learning rates are *DIFFERENT* for different connections, ordinary backprop given unstable rates will teach the network as a whole to use nodes fed by connections with lower learning rates, which are still stable - and in so doing downregulate the error attributable to the unstable connections meaning they get less training and become stable as a side effect. Conversely, if all connections are stable, backprop rapidly drives the later nodes to use the highest-learning-rate connections.
The *effective* learning rate of both weights combined, given weights with learning rates a "reasonable" factor different from each other, winds up balanced at just about the maximum it can go to while remaining stable, provided it's higher than the lowest available rate and lower than the highest available rate. And you can cover a huge dynamic range with an exponential distribution by having about eight different learning rates each of which is about a quarter of the one before it.
A solution to Catastrophic Forgetting falls out because the first task is encoded in low-rate connections. It doesn't get forgotten because whatever it *doesn't* have in common with the second task gets the later connections that depend on it rapidly downregulated until training just about isn't happening at all. Conversely whatever it *does* have in common with the second task continues to get trained, and is also useful in combination with the training preserved in the very low-rate nodes when switching back to the first one. If you switch between tasks occasionally (once a day or once a week) as far as the lowest-rate nodes are concerned they're getting trained on whatever is common to both and, if not, they effectively don't get trained during the times when they're "inapplicable" to the current task.
The problem was that in conventional neural network architecture all the connections have the *SAME* learning rate. If the learning rates are *DIFFERENT* for different connections, ordinary backprop given unstable rates will teach the network as a whole to use nodes fed by connections with lower learning rates, which are still stable - and in so doing downregulate the error attributable to the unstable connections meaning they get less training and become stable as a side effect. Conversely, if all connections are stable, backprop rapidly drives the later nodes to use the highest-learning-rate connections.
The *effective* learning rate of both weights combined, given weights with learning rates a "reasonable" factor different from each other, winds up balanced at just about the maximum it can go to while remaining stable, provided it's higher than the lowest available rate and lower than the highest available rate. And you can cover a huge dynamic range with an exponential distribution by having about eight different learning rates each of which is about a quarter of the one before it.
A solution to Catastrophic Forgetting falls out because the first task is encoded in low-rate connections. It doesn't get forgotten because whatever it *doesn't* have in common with the second task gets the later connections that depend on it rapidly downregulated until training just about isn't happening at all. Conversely whatever it *does* have in common with the second task continues to get trained, and is also useful in combination with the training preserved in the very low-rate nodes when switching back to the first one. If you switch between tasks occasionally (once a day or once a week) as far as the lowest-rate nodes are concerned they're getting trained on whatever is common to both and, if not, they effectively don't get trained during the times when they're "inapplicable" to the current task.