Automatic Color Scheme Extraction from Movies
Automatic Color Scheme Extraction from Movies
Suzi Kim and Sunghee Choi
Geometric Computing Lab., School of Computing, KAIST
Presented at 2020 ACM International Conference on Multimedia Retrieval (ICMR)
Abstract
A color scheme is an association of colors, i.e., a subset of all possible colors, that represents a visual identity. We propose an automated method to extract a color scheme from a movie. Since a movie is a carefully edited video with different objects and heterogeneous content embodying the director's messages and values, it is a challenging task to extract a color scheme from a movie as opposed to a general video filmed at once without distinction of shots or scenes. Despite such challenges, color scheme extraction plays a very important role in film production and application. The color scheme is an interpretation of the scenario by the cinematographer and it can convey a mood or feeling that stays with the viewer after the movie has ended. It also acts as a contributing factor to describe a film, like the metadata fields of a film such as a genre, director, and casting. Moreover, it can be automatically tagged unlike metadata, so it can be directly applied to the existing movie database without much effort. Our method produces a color scheme from a movie in a bottom-up manner from segmented shots. We formulate the color extraction as a selection problem where perceptually important colors are selected using saliency. We introduce a semi-master-shot, an alternative unit defined as a combination of contiguous shots taken in the same place with similar colors. Using real movie videos, we demonstrate and validate the plausibility of the proposed technique.
How to Run
Our system consists of three modules: semi-master-shot boundary detection (SBD), base palette extraction (BPE), and color scheme merge (CSM). Unfortunately, three modules are not integrated together, so you should run the module step by step.
| Directory | Language | Outputs | |
|---|---|---|---|
| SBD | 1-SBD | C++ | txt file including shot boundary |
| BPE | 2-BPE | Matlab | base palettes |
| CSM | 3-CSM | Python | color scheme images and hex file |
Installation
- Clone this repo:
$ cd ICMR2020-MovieColorSchemer
Dataset
We evaluate our method on two datasets: OVSD and a commercial movie dataset. The movies were encoded at 24 fps, resized 320 pixels wide, and opening and closing credits were discarded. Download the dataset and place them under the ICMR2020-MovieColorSchemer directory as follows:
+-- videos
+-- CMD
| +-- atonement.avi
| +-- forrest_gump.avi
| +-- ...
| +-- life_of_pi.avi
+-- OSVD
+-- la_chute_dune_plume.avi
+-- elephants_dream.avi
+-- ...
+-- tears_of_steel.avi
(1) OVSD
- Open Video Scene Detection Dataset (OVSD)
- OVSD is an open dataset of Creative Commons licensed videos freely available for download and use, comprising 21 short or full-length movies from various genres, e.g. drama, animation, crime, sci-fi, etc. OVSD is designed for scene detection, and the composition of the movie is aesthetically insufficient to evaluate the color scheme extraction.
(2) Commercial Movie Dataset (CMD)
To compare the final color scheme results, we use a commercial movie dataset instead of the OVSD. Although movies contain richer narrative patterns of shots and scenes compared with those general videos, there is no dataset targeting commercial movies due to copyright issues. We collect the commercial movie dataset particularly to compare current works by artists or descriptor roles. The dataset contains 53 movies from various genres, including Inception, Atonement, Charlie and the Chocolate Factory, Days of Being Wild, Forrest Gump, Eternal Sunshine of the Spotless Mind, Her, La La Land, Life of Pi, Mary Poppins, Melancholia, Oldboy, Punch-Drunk Love, Spirited Away, The Avengers, The Grand Budapest Hotel, The Matrix, Your Name, etc.
Prerequisite
- Visual Studio 2013 or higher (We tested on the Visual Studio 2019)
- Platform toolset to Visual Studio 2013 (v120)
- CUDA, cuDNN, Theano
- We tested on CUDA 9.1 and cuDNN v.7.0.5 which are compatible with Theano 1.0.4.
(1) Semi-master-shot Boundary Detection (SBD)
We adopt the Imagelab Shot Detector (ILSD) segmentation method, which only considers RGB colors.
We acquire semi-master-shots mitigating the threshold to determine the color difference between shots. To enhance the function of the safe zone, we use , which is proportional to the average length of shots, instead of the fixed value of .
We added three functions to the ILSD code.
- making it get an additional parameters:
- rather than the fixed , calculating according to the average shot length by the original .
- generating a subtitle file (.srt) to check the segmentation results visually.
We only modified the main.cpp file and left the comment with the tag 'ICMR2020-Suzi' in the code where we modified from the original ILSD.
You can build '1-SBD' directory on the Visual Studio with 'Release x64' configuration, and it generates 'SBD.exe' binary file. Then, to get a semi-master-shot boundary information,
and it generates a shot boundary file at results/semi-master-shots/
start_frame_of_shot#1 end_frame_shot#1
start_frame_of_shot#2 end_frame_shot#2
...
(2) Base Palette Extraction (BPE)
BPE gets two inputs: (1) a video file and (2) semi-master-shot file, which are acquired from the SBD stage.
To calculate the saliency map, we adopt the SalGAN. Download the two pretrained models VGG-16 weights(vgg16.pkl) and SalGAN generator model(gen_modelWeights0090.npz), and put them under saliency-salgan-2017/ directory.
The whole process of the BPE is done by following code:
$ sudo matlab -nojvm -nodesktop -nosplash
>> setenv LD_LIBRARY_PATH matlabroot/bin/glnxa64:matlabroot/sys/os/glnxa64
>> addpath '2-BPE'
>> BPE('OVSD', 'la_chute_dune_plume.avi')
