Week 8: Generating Images with Neural Style Transfer

A popular machine learning method for generation images is style transfer: appropriating the style of one image onto the content of another. Here is jcjohnson’ neural-style model, for which there is excellent documentation and examples.

I cloned the model into my virtual GPU machine in Paperspace (described in this post), and experimented with it in two ways: traditional style transfer and texture synthesis.

PART 1 - STYLE TRANSFER
This technique reminds of me of layering images with masks in Photoshop. To run the model you select and train on a “style” image to apply to a “content” image. There are many adjustable parameters to optimize the process and that impact the look of the resulting image, all documented in the repo.

Here’s an example (command run inside of model’s directory):
$ th neural_style.lua -style_image /home/paperspace/Documents/projects/_pattern.jpg -content_image /home/paperspace/Documents/projects/_portrait.jpg -output_image /home/paperspace/Documents/projects/result/_pattern_portrait.png -style_weight 2e0 -content_weight 3e0 -backend cudnn

My style image is of a pattern, my content image (to receive the style) is a portrait, I’ve defined the location and name of the resulting image, I’ve adjusted how much to weigh the style and content inputs, and finally, made more efficient use of my GPU’s memory by using the -backend cudnn flag. (Channel Zero, anyone?)

Let’s try another example, this time with the default style and content weight settings of 5e0 and 1e2 respectively. It really is like being in an analog dark room. No two prints are alike. Even if you run the model again with the same images and parameters, you get slightly different results.

PART 2 - TEXTURE SYNTHESIS
When you run the model with the content weight equal to 0, it still picks up and applies the learned style to an empty canvas. Again the result changes with every run even if the parameters do not change.

$ th neural_style.lua -style_image /home/paperspace/Documents/projects/clouds.jpg -content_image /home/paperspace/Documents/projects/content.jpg -output_image /home/paperspace/Documents/projects/result/_clouds_texture.png -content_weight 0 -backend cudnn

You can also combine multiple style input images for a collaged-texture synthesis:

$ th neural_style.lua -style_image /home/paperspace/Documents/projects/one.jpg,/home/paperspace/Documents/projects/two.jpg -content_image /home/paperspace/Documents/projects/content.jpg -output_image /home/paperspace/Documents/projects/result/_trees_texture.png -content_weight 0 -backend cudnn

Image Credits: Clouds & Red, Blue, and White Abstract

I guess this okay; I’m still looking for a useful application of this (because Photoshop). But I’m glad I did it. And really I want to know how it handles many many many input style images—like all 6,555 images from my Blade Runner experiment earlier. Unfortunately, Terminal said my argument list was too long.

I then tried half that amount (having extracted 30 frames from each minute of the film) but again got the same response.

It does work with 26 pics, though! Which is what I used when I tested the process (though I already trashed that output image). My next step is to figure out how to get around this…

In the mean time, here are the steps I used to prepare to train the model on multiple style files; I concatenated all the filenames with a comma, stored that into giant string which I wrote to a text file, then stored the contents of that file into a variable, which I then included in the command to start the training process:

  1. Create an empty text file: $ touch text.txt

  2. Create a directory with your image files

  3. Start a python shell: $ python

  4. >>> import os

  5. >>> mydir = ‘/images’

  6. >>> style_string = ','.join([os.path.join(mydir, f) for f in os.listdir(mydir)])

  7. >>> f = open( 'file.txt’, 'w' )

  8. >>> f.write(style_string)

  9. >>> f.close()

  10. Exit python shell (Control + d)

  11. Navigate into neural style directory

  12. $ value=$(</dir/file.txt)

  13. $ echo “$value” (to check the contents of the variable)

Which then leads to this:
$ th neural_style.lua -style_image "$value" -content_image /home/paperspace/Documents/projects/content.jpg -output_image /home/paperspace/Documents/projects/result/_result.png -content_weight 0 -backend cudnn