The system, called " GANpaint Studio ," allows a user to upload an image of their choosing and modify multiple aspects of its appearance, from changing the size of objects to adding completely new items like trees and buildings. In addition to helping artists and designers make quick adjustments to visuals, say the researchers, the system may help computer scientists identify "fake" images.
Adapting the system to video clips could enable computer-graphics editors to quickly compose specific arrangements of objects needed for a particular shot. GANpaint Studio, say the researchers, could also be used to improve and debug other GANs that are being developed, by analyzing them for "artifact" units that need to be removed and help researchers better understand neural networks and their underlying structures.
"Right now, machine learning systems are these black boxes that we don't always know how to improve, kind of like those old TV sets that you have to fix by hitting them on the side, says David Bau, a PhD student at MIT's Computer Science and Artificial Intelligence Lab (CSAIL)and lead author on a related paper. "This research suggests that, while it might be scary to open up the TV and take a look at all the wires, there's going to be a lot of meaningful information in there."
An unexpected discovery, say the researchers, is that the system seems to have learned some simple rules about the relationships between objects - it knows not to put something somewhere it doesn't belong, like a window in the sky, and it also creates different visuals in different contexts. For example, if there are two different buildings in an image and the system is asked to add doors to both, it doesn't simply add identical doors - they may ultimately look quite different from each other.
"All drawing apps will follow user instructions, but ours might decide not to draw anything if the user commands to