in Machine Learning, Programming, Python, Tutorial

Generating Deep Dreams

The previous post in this series, Understanding Deep Dreams, explained what deep dreams are, and what they can be used for. In this second post you’ll learn how to create them, with a step by step guide.

Introduction

In August 2015, Google Research released the full source code they have been using to generate their pictures. My guide is strongly inspired by this one, which unfortunately didn’t work for many users. The following instructions will show how to virtualise Ubuntu 14.04.02 with VirtualBox, and how to run your very own deep dreams on it.

You can test this very technique using DeepDreamThis, a twitter bot that will create deep dreams out of the images you send to it. You can check a detailed guide on how to use it here. You can also check DeepForger, which uses neural networks to transfer artistic styles from a painting to a picture.

The Instructions

  • Virtualise Ubuntu 14.04.02
    • Install VirtualBox
    • Download the Ubuntu 14.04.02 iso
    • Create a new Virtual machine for “Linux / Ubuntu 64bit“.
    • You’ll need at least 10Gb of space and at least 2048Mb of RAM
    • In the Settings > Storage properties, add the downloaded iso of Ubuntu in the CD drive
    • Run the virtual machine
    • Install Ubuntu
  • Install Guest Additions
    • From the VirtualBox windows: Device > “Insert Guest Additions CD image
    • Install
    • Restart the machine
  • Install build essentials
    • sudo apt-get install build-essential
    • sudo apt-get install linux-headers-`uname -r`
  • Install caffe and deepdream dependencies
    • sudo apt-get install -y curl libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev protobuf-compiler gfortran libjpeg62 libfreeimage-dev libatlas-base-dev git python-dev python-pip libgoogle-glog-dev libbz2-dev libxml2-dev libxslt-dev libffi-dev libssl-dev libgflags-dev liblmdb-dev python-yaml imagemagick python-opencv
    • sudo easy_install pillow
  • Download caffe
    • cd ~
    • git clone https://github.com/BVLC/caffe.git
  • Install caffe python dependencies
    • cd caffe
    • cat python/requirements.txt | xargs -L 1 sudo pip install
    • sudo ln -s /usr/include/python2.7/ /usr/local/include/python2.7
    • sudo ln -s /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ /usr/local/include/python2.7/numpy
  • Configure caffe
    • cp Makefile.config.example Makefile.config
    • nano Makefile.config
    • Uncomment the line # CPU_ONLY := 1
    • Under PYTHON_INCLUDE, replace /usr/lib/python2.7/dist-packages/numpy/core/include with /usr/local/lib/python2.7/dist-packages/numpy/core/include
  • Compile caffe
    • make pycaffe
    • make all
    • make test
    • make runtest
    • Be sure the output of make runtest doesn’t include any error
    • make distribute
  • Make pycaffe available to python
    • sudo cp distribute/lib/libcaffe.so distribute/python/caffe
    • sudo cp -r distribute/python/caffe /usr/local/lib/python2.7/dist-packages/caffe
    • sudo nano /etc/ld.so.conf.d/caffe.conf
    • LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/caffe/distribute/python/caffe
      Write the full path to the caffe/distribute/lib/  folder; for instance: /home/alanzucconi/caffe/distribute/lib/ . Do not use ~.
    • sudo ldconfig
    • You should also put these lines in the .bashrc file, so that they are executed when your computer starts (remember to use the path to your folders):
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/caffe/distribute/python/caffe
      export PYTHONPATH=$PYTHONPATH:~/caffe/distribute/python/
  • Download the GoogLeNet model
    • cd ~/caffe/models/bvlc_googlenet
    • curl -O "http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel"
  • Download deepdream
    • cd ~
    • git clone https://github.com/google/deepdream.git
  • The code
    • In the deepdream folder create a file called dream.py
    • Fill it with the code below
    • cd ~/deepdream
    • sudo python dream.py

The code

# imports and basic notebook setup
from cStringIO import StringIO
import numpy as np
import scipy.ndimage as nd
import PIL.Image
from IPython.display import clear_output, Image, display
from google.protobuf import text_format
import cv2
import caffe

showimages = True
def showarray(a, fmt='jpeg'):
    a = np.uint8(np.clip(a, 0, 255))
    f = StringIO()
    PIL.Image.fromarray(a).save(f, fmt)
    global showimages
    if showimages:
        a = cv2.cvtColor(a, cv2.COLOR_BGR2RGB)
        cv2.imshow('image',a)
        cv2.waitKey(1)

model_path = '../caffe/models/bvlc_googlenet/' # substitute your path here
net_fn   = model_path + 'deploy.prototxt'
param_fn = model_path + 'bvlc_googlenet.caffemodel'

# Patching model to be able to compute gradients.
# Note that you can also manually add "force_backward: true" line to "deploy.prototxt".
model = caffe.io.caffe_pb2.NetParameter()
text_format.Merge(open(net_fn).read(), model)
model.force_backward = True
open('tmp.prototxt', 'w').write(str(model))

net = caffe.Classifier('tmp.prototxt', param_fn,
                       mean = np.float32([104.0, 116.0, 122.0]), # ImageNet mean, training set dependent
                       channel_swap = (2,1,0)) # the reference model has channels in BGR order instead of RGB

# a couple of utility functions for converting to and from Caffe's input image layout
def preprocess(net, img):
    return np.float32(np.rollaxis(img, 2)[::-1]) - net.transformer.mean['data']
def deprocess(net, img):
    return np.dstack((img + net.transformer.mean['data'])[::-1])

def make_step(net, step_size=1.5, end='inception_4c/output', jitter=32, clip=True):
    '''Basic gradient ascent step.'''

    src = net.blobs['data'] # input image is stored in Net's 'data' blob
    dst = net.blobs[end]

    ox, oy = np.random.randint(-jitter, jitter+1, 2)
    src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift
            
    net.forward(end=end)
    dst.diff[:] = dst.data  # specify the optimization objective
    net.backward(start=end)
    g = src.diff[0]
    # apply normalized ascent step to the input image
    src.data[:] += step_size/np.abs(g).mean() * g

    src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image
            
    if clip:
        bias = net.transformer.mean['data']
        src.data[:] = np.clip(src.data, -bias, 255-bias)    

def deepdream(net, base_img, iter_n=10, octave_n=4, octave_scale=1.4, end='inception_4c/output', clip=True, **step_params):
    # prepare base images for all octaves
    octaves = [preprocess(net, base_img)]
    for i in xrange(octave_n-1):
        octaves.append(nd.zoom(octaves[-1], (1, 1.0/octave_scale,1.0/octave_scale), order=1))
    
    src = net.blobs['data']
    detail = np.zeros_like(octaves[-1]) # allocate image for network-produced details
    for octave, octave_base in enumerate(octaves[::-1]):
        h, w = octave_base.shape[-2:]
        if octave > 0:
            # upscale details from the previous octave
            h1, w1 = detail.shape[-2:]
            detail = nd.zoom(detail, (1, 1.0*h/h1,1.0*w/w1), order=1)

        src.reshape(1,3,h,w) # resize the network's input image size
        src.data[0] = octave_base+detail
        for i in xrange(iter_n):
            make_step(net, end=end, clip=clip, **step_params)
            
            # visualization
            vis = deprocess(net, src.data[0])
            if not clip: # adjust image contrast if clipping is disabled
                vis = vis*(255.0/np.percentile(vis, 99.98))
            showarray(vis)
            print octave, i, end, vis.shape
            clear_output(wait=True)
            
        # extract details produced on the current octave
        detail = src.data[0]-octave_base
    # returning the resulting image
    return deprocess(net, src.data[0])

img = np.float32(PIL.Image.open('sky1024px.jpg'))
showarray(img)

frame=deepdream(net, img)
PIL.Image.fromarray(np.uint8(frame)).save("frames/a.jpg")

frame=deepdream(net, img, end='inception_3b/5x5_reduce')
PIL.Image.fromarray(np.uint8(frame)).save("frames/b.jpg")

frame = img
frame_i = 0
h, w = frame.shape[:2]
s = 0.05 # scale coefficient
for i in xrange(100):
    frame = deepdream(net, frame)
    PIL.Image.fromarray(np.uint8(frame)).save("frames/%04d.jpg"%frame_i)
    frame = nd.affine_transform(frame, [1-s,1-s,1], [h*s/2,w*s/2,0], order=1)
    frame_i += 1

cv2.destroyAllWindows()

Lines 98 and 101 uses the deepdream function to generate different dreams. The end parameter is used to determine which layer we want to analyse. You can check which layers can be analysed by printing net.blobs.keys().

deepdream

I’ve created a repository of all the layers you can query. Find an effect you like, and use the name of the file (without the extension) and the end parameter. Just remember to replace any __ (double underscore) with a / (Unix slash).

Conclusion

This post explained how to use the machine learning framework Caffe to generate deep dreams. Using a virtualised environment, however, might not be the best option. Neural networks require a lot of computational power, which is often unavailable in a virtual machine. For best performances, you should run deep dreams using a GPU.

This tutorial referred to the first, original tutorial released by Google Research. Other techniques have been evolved in the past months, including algorithms that works on videos.

Other resources

  • deepdream: the GIT project used by Google Research;
  • Caffe: the framework used to implement neural networks. Valid alternatives are Theano and Torch;
  • Running the deep dream: an alternative tutorial to run deep dreams;
  • neural-style: a Torch variation of deep dreams, used to transfer artistic styles;
  • neural-doodle: a tool that allows to draw in the style of a particular artist.
💖 Support this blog

This website exists thanks to the contribution of patrons on Patreon. If you think these posts have either helped or inspired you, please consider supporting this blog.

Patreon Patreon_button
Twitter_logo

YouTube_logo
📧 Stay updated

You will be notified when a new tutorial is released!

📝 Licensing

You are free to use, adapt and build upon this tutorial for your own projects (even commercially) as long as you credit me.

You are not allowed to redistribute the content of this tutorial on other platforms, especially the parts that are only available on Patreon.

If the knowledge you have gained had a significant impact on your project, a mention in the credit would be very appreciated. ❤️🧔🏻

Write a Comment

Comment

  1. Hi Alan,
    thanks for this – I think there are a couple of errors:

    under Make pycaffe available to python
    sudo nano /etc/ ld.so.conf.d/caffe.conf
    — there is a space between /etc/ and ld.so….

    and in your code sample:
    line 17 i f showimages:

    I get errors when I try to import caffe in python. Trying to figure that out and will keep you posted.

    thanks!

    • Hi Jan! Believe it or not, those mistakes were intentional.
      At the time I posted the article there was a security problem with the server. Basically it wasn’t loading pages with certain pieces of code it believed to be “dangerous”. For some reason those two lines were considered suspicious. The issue is not solved (for now!) for I forgot to fix them! 😀

      Thank you! 😀

        • Hi, I know it’s a while, but maybe you could still help me out here.
          I’ve encountered the same problem as mika. I changed the ~ into /home/mat/ but that doesn’t do it. If you or anyone has answer, it would be most welcome. Thanks
          Mat

  2. When I switch to gpu mode, I get this error:
    F1129 00:06:07.387610 31472 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
    *** Check failure stack trace: ***
    This happens in both linux and windows .
    The irony is it happens randomly, meaning for some it just happens at the very begining of the process, for some others it happens at the very end.
    Did you encounter this? or do you have any idea why this is happening and how I could fix it?
    Thanks in advance

    • Hey!

      If you’re using VirtualBox, you can’t access the GPU.
      Did you check if your GPU is actually supported by CUDA?

      You say this is happening randomly. When it works, do you actually get a substantial speed up?

  3. I encountered a compiler error running make missing libhdf5.h on ubuntu 16.04 LTS. I fixed it adding

    INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/ and
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/

    to makefile should anyone encounter this error.

    source: https://github.com/NVIDIA/DIGITS/issues/156

Webmentions

  • Understanding Deep Dreams – Alan Zucconi November 14, 2019

    […] Part 2. Generating Deep Dreams […]