Image classification with Keras CV

Introduction 

In this walkthrough, we will explore image classification by leveraging Keras CV & the EfficientNet model.

For those unfamiliar with Keras, it is a multi-framework API supporting JaxTensorflow & PyTorch as backends. Keras CV can be thought of as a Keras extension specific to computer vision related tasks. 

Warming up for image classification

Let’s start by setting up a new Python virtual environment to contain all our project dependencies, the process is quite simple: 

(A) Create a new folder to contain your project 

(B) Open a terminal, go into your new folder and run: 


gweisz folder_labs % python3 -m venv . 
(Notice that everything before the “%” character is my system prompt) 

Now that the new environment is set-up it’s worth mentioning some key commands: 

  • If you want to start working in the virtual environment, position the terminal within the project folder and type: 

gweisz folder_labs % source ./bin/activate

 

  • You will notice that the terminal prompt changes to contain your virtual env name between parenthesis, e.g.: 

gweisz folder_labs % source ./bin/activate 
(folder_labs) gweisz folder_labs %

 

  • At every time you can leave a virtual environment by running: 

 

(folder_labs) gweisz folder_labs % deactivate
gweisz folder_labs
 

  • Finally, if you ever want to delete the virtual env and all its content deactivate it and delete the folder 
 

We will now proceed to install the required dependencies, but before doing so it’s usually worth checking if the pip itself can be upgraded 

(folder_labs) gweisz folder_labs % python -m pip install --upgrade pip 
 

The following are the dependencies we will need for the walkthrough: 

(folder_labs) gweisz folder_labs % pip install --upgrade tensorflow 
(folder_labs) gweisz folder_labs % pip install --upgrade keras 
(folder_labs) gweisz folder_labs % pip install --upgrade keras-cv 
(folder_labs) gweisz folder_labs % pip install pillow 
(folder_labs) gweisz folder_labs % pip install matplotlib 
 

The dataset 

Let’s start by putting in place the dataset of moon & sun images. I just assembled one taking pictures from Unsplash, if you want to assemble your own using different pictures and/or on a different topic go ahead, but beware that some licenses forbid AI usage (e.g. Unsplash+ https://unsplash.com/plus/license). 

The complete dataset can be found within: https://github.com/gweisz-folder/sun-moon-demo/tree/main/datasets/sun_moon, particularly see ‘train’ & ‘validation’ sub-folders. 

This is the perfect time to decide the sizing of the train and the validation splits. Personally, and to demonstrate the idea, I’m going with 20 images in the train split (10 suns + 10 moons) and 10 images in the validation split (5 & 5), but be mindful that this is usually a hard topic, just to expand on this fact, the following are a couple of variations that should be considered for such dataset: 

  • Sun: 
    • Include sun pictures at different moments of the day (e.g. Early morning vs noon vs late afternoon) 
    • Include sun pictures with clouds (consider different types of clouds) 
    • Include sun pictures in different places & seasons (e.g. snowy woods vs temperate grasslands, etc.) 
    • Combinations of all the above!
  • Moon:
    • Then repeat the same mental process for the moon
    • There are also full moons, half-moons, etc. 
    • Finally, the moon can be seen during both the day & the night 

You might be wondering whether (for a real dataset & classification task) such attention to detail is required, long story short, yes. For example, if we omit the last point, the model might just learn to distinguish day from night… 

One final thing before we start assembling the dataset ‘programmatically’, let’s rename each picture such that the filename is {class}-{author}.{extension} to ease the upcoming steps, with potential classes being either ‘sun’ or ‘moon’. 

Example: 

“bechir-kaddech-Gqz1hSB-iiU-unsplash.jpg” will turn into “moon-bechir-kaddech.jpg” 

 At this point we are ready to assemble our dataset, let’s now leverage the tensorflow_datasets module. I recommend you create a new “datasets” subfolder, then within the virtual env, move to such subfolder and initialize the new dataset with the following command: 

(folder_labs) gweisz datasets % tfds new sun_moon 

Notice how this created a new subfolder plus a lot of automatically generated code. Let's now create two sub-folders, one for each split named ‘train’ & ‘validation’. Now pick 10 of the sun & 10 of the moons pictures for the train split and move them to their subfolder, and then the same thing for the rest of them into the validation split. 

Although tfds is easy to use, it’s certainly not the easiest thing to learn if you haven’t used it before, if you’d like to get yourself a little more familiar with it before moving on, I recommend you review tfds add_dataset. 

Now it’s time to start adapting the automatically generated code, let’s start with the simple stuff. 

TAGS.txt: 

Likely just: 

  • content.data-type.image # Contains image data. 
  • content.subject.earth-and-nature # Relates to earth and nature. 
  • ml.task.image-classification # Relates to Image Classification, a machine learning task. 

Delete all other tags. 

README.md 

Just a markdown file to describe the dataset, let’s do something like this: 

# Title 

 Sun & Moon dataset 
 
# Description 
 

Demonstration dataset used for a technical blog @[Folder IT](<https://folderit.net/>) 

 # Data acquisition 
 
All pictures were taken from [Unsplash](<https://unsplash.com/>) 

# How to build: 

 `tfds build` 

 

CITATIONS.bib 

No citation needed 

 

checksums.tsv 

Nothing needed here 

 

init.py 

Just leave empty 

 

dummy_data/ 

No need to do anything in this subfolder 

 

sun_moon_dataset_builder_test.py 

Just leave it as is 

 

sun_moon_dataset_builder.py 

We have finally reached the place where we will define & assemble our dataset! 

_info method: Here we need to state the dataset metadata, which for us can look like this: 

				
					def _info(self) -> tfds.core.DatasetInfo: 
    """Returns the dataset metadata.""" 
    return self.dataset_info_from_configs( 
        features=tfds.features.FeaturesDict({ 
            'image': tfds.features.Image(shape=(None, None, 3)), 
            'filename': tfds.features.Text(), 
            'label': tfds.features.ClassLabel(names=['sun', 'moon']), 
        }), 
        supervised_keys=('image', 'label'), 
    ) 
				
			

_split_generators method: We won’t be downloading the data, but assemble the dataset with the downloaded images, accordingly this one will be pretty straightforward, like: 

				
					def _split_generators(self, dl_manager): 
    """Returns SplitGenerators.""" 
    return { 
        'train': self._generate_examples('train'), 
        'validation': self._generate_examples('validation'), 
    }
				
			

_generate_examples method: Here we’ll leverage the local pictures we renamed within the train & validation sub-folders, like: 

				
					# Add these imports above
import os
from PIL import Image

# The method
def _generate_examples(self, path): 
    """Yields examples.""" 
    for f in os.listdir(path): 
        label = '' 
 
        if f.startswith('sun-'): 
            label = 'sun' 
        elif f.startswith('moon-'): 
            label = 'moon' 
        else: 
            raise Exception("Unrecognized file") 
         
        if not f.endswith('.jpg'): 
            raise Exception("Unexpected file extension") 
 
        with Image.open(os.path.join(path, f)) as image: 
            yield f, { 
                'image': image, 
                'filename': f, 
                'label': label, 
            } 
				
			

At this point we are ready to build the dataset, just go into the dataset folder and run: 

(folder_labs) gweisz sun_moon % tfds build

This command will effectively build the dataset & leave it prepared within ~/tensorflow_datasets/ 

Data loading & preview 

Now let’s start to work on the main logic, for this purpose let’s create a file named classifier.py in the project main folder, let’s try loading and previewing the images from the validation split along with their related label. This can be achieved with something like: 

				
					# Imports 
import os 
import tensorflow_datasets as tfds 
import keras_cv 
import matplotlib.pyplot as plt 
import matplotlib.patches as patches 
 
os.environ["KERAS_BACKEND"] = "tensorflow" 
 
#  Main parameters 
DS_NAME = 'sun_moon' 
 
VALIDATION_BATCH_SIZE = 1 
 
# Load the DS 
validation_ds = tfds.load(DS_NAME, split='validation', as_supervised=True).batch( 
    VALIDATION_BATCH_SIZE).prefetch(1) 
 
# Lets review each image along with its label 
for tensors in validation_ds: 
    image, label = tensors[0], tensors[1] 
 
    keras_cv.visualization.plot_image_gallery(image, value_range=( 
        0, 255), legend_handles=[patches.Patch(label=str(label))]) 

    # The following line is only required if you are running this on your local computer, on environments like Google Colab it isn't 
    # It will wait until the user closes the picture to continue 
    plt.show()
				
			


Example:

Notice that both the image and its label are tensors under the hood, for that reason I explicitly added the label as a tensor in the legend, typically a value of [0] would represent the sun, and [1] the moon. 

If you want to check the image tensor you can just print it out to the terminal, e.g. 

It’s important to understand that for batched datasets, the shape of the image tensor is (batch size, height, width, channels).

Data Pre-processing 

Now that we have an understanding of the underlying tensors it’s time to move ahead with the data preprocessing, in this phase, we will resize all images into 224 by 224 in a consistent manner utilizing the Keras CV resizing layer & turn the label into a one hot encoding. Finally let’s see how the pictures look after this transformation: 

				
					# Imports 
import os 
import tensorflow as tf 
import tensorflow_datasets as tfds 
import keras_cv 
import matplotlib.pyplot as plt 
import matplotlib.patches as patches 
 
os.environ["KERAS_BACKEND"] = "tensorflow" 
 
#  Main parameters 
DS_NAME = 'sun_moon' 
VALIDATION_BATCH_SIZE = 4 
IMAGE_TARGET_SIZE = (224, 224) 
NUM_CLASSES = 2 
 
# Functions 
 
resizing_f = keras_cv.layers.Resizing( 
    IMAGE_TARGET_SIZE[0], IMAGE_TARGET_SIZE[1], crop_to_aspect_ratio=True 
) 
 
def preprocess_data(images, labels): 
    return resizing_f(images), tf.one_hot(labels, NUM_CLASSES) 
 
# Load the DS 
validation_ds = tfds.load(DS_NAME, split='validation', as_supervised=True).map( 
    preprocess_data, num_parallel_calls=tf.data.AUTOTUNE).batch(VALIDATION_BATCH_SIZE).prefetch(2) 
 
# Lets review each image along with its label 
for tensors in validation_ds: 
    image, label = tensors[0], tensors[1] 
 
    keras_cv.visualization.plot_image_gallery(image, value_range=( 
        0, 255), legend_handles=[patches.Patch(label=str(label))]) 

    # The following line is only required if you are running this on your local computer, on environments like google Colab it isn't
    # It will wait until the user closes the picture to continue
    plt.show()
				
			

 

As we can see now the pictures have an equal size, and ‘sun’ is now represented as [1, 0] while ‘moon’ as [0, 1] which is exactly what we expected. Since we are using “crop_to_aspect_ratioit’s important to review that the sun/moon is visible in all images (This applies also to the train split).

Data augmentation 

Now it’s time to also incorporate some data augmentation layers! Let’s just do some simple augmentation by leveraging a RandomRotation layer, and a RandomFlip layer 

Just define the following set of layers in the functions section. 

				
					data_augmentation = keras.Sequential( 
    [ 
        keras.layers.RandomRotation(factor=0.1), 
        keras.layers.RandomFlip("horizontal"), 
    ] 
)

# And modify the data pipeline to look like: 

validation_ds = tfds.load(DS_NAME, split='validation',  as_supervised=True).map(
    preprocess_data).map(lambda x, y: (data_augmentation(x),  y)).batch(VALIDATION_BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
				
			
 

Image Classification 

We are now ready to pick our classification model, for the task at hand let’s just leverage a Keras CV ImageClassifier, particularly let’s use the “efficientnetv2_b0_imagenet” preset 

Start by loading the model and producing a summary of it, just add: 

				
					model = keras_cv.models.ImageClassifier.from_preset(
    "efficientnetv2_b0_imagenet", num_classes=NUM_CLASSES
)

model.summary() 
				
			

 

As you can see the model’s output layer has already been changed to only predict two classes: 

We now need to decide how to train our new model, although several options exist, for this walkthrough let’s do some simple fine-tuning.

That is, we will perform training in two phases: 

(Phase A) Here we will train the latest layer only, leveraging the pre-existent network weights for all the previous layers. 

(Phase B) Then we will allow the whole network to be trainable and train for a few epochs to see if this has a positive effect in general. 

We will also plot loss & metrics against epochs for each phase, so let’s revisit our code to achieve all this.

				
					# Imports 
import os 
import tensorflow as tf 
import tensorflow_datasets as tfds 
import keras 
import keras_cv 
import matplotlib.pyplot as plt 
import matplotlib.patches as patches 
 
os.environ["KERAS_BACKEND"] = "tensorflow" 
 
#  Main parameters 
DS_NAME = 'sun_moon' 
BATCH_SIZE = 4 
VALIATION_BATCH_SIZE = 10 
IMAGE_TARGET_SIZE = (224, 224) 
NUM_CLASSES = 2 
LEARNING_RATE = 0.02 
EPOCHS_A = 50 
EPOCHS_B = 10 
 
# Functions 
resizing_f = keras_cv.layers.Resizing( 
    IMAGE_TARGET_SIZE[0], IMAGE_TARGET_SIZE[1], crop_to_aspect_ratio=True 
) 
 
def preprocess_data(images, labels): 
    return resizing_f(images), tf.one_hot(labels, NUM_CLASSES) 
 
data_augmentation = keras.Sequential( 
    [ 
        keras.layers.RandomRotation(factor=0.1), 
        keras.layers.RandomFlip("horizontal"), 
    ] 
) 
 
def graph_history(history): 
    plt.plot(history.history['accuracy']) 
    plt.plot(history.history['val_accuracy']) 
    plt.title('model accuracy') 
    plt.ylabel('accuracy') 
    plt.xlabel('epoch') 
    plt.legend(['train', 'val'], loc='upper left') 
    # Wait until the user closes the graph 
    plt.show() 
 
    plt.plot(history.history['loss']) 
    plt.plot(history.history['val_loss']) 
    plt.title('model loss') 
    plt.ylabel('loss') 
    plt.xlabel('epoch') 
    plt.legend(['train', 'val'], loc='upper left') 
    # Wait until the user closes the graph 
    plt.show() 
 
# Dataset & splits 
train_ds = tfds.load(DS_NAME, split='train', as_supervised=True).map( 
    preprocess_data).map(lambda x, y: (data_augmentation(x), y)).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) 
 
validation_ds = tfds.load(DS_NAME, split='validation', as_supervised=True).map( 
    preprocess_data).batch(VALIATION_BATCH_SIZE).prefetch(tf.data.AUTOTUNE) 
 
# Our classification model 
model = keras_cv.models.ImageClassifier.from_preset( 
    "efficientnetv2_b0_imagenet", num_classes=NUM_CLASSES 
) 
 
# Phase A 
model.get_layer('efficient_net_v2b0_backbone').trainable = False 
model.summary() 
 
model.compile( 
    loss="categorical_crossentropy", 
    optimizer=keras.optimizers.SGD(learning_rate=LEARNING_RATE), 
    metrics=["accuracy"], 
) 
 
history = model.fit( 
    train_ds, 
    validation_data=validation_ds, 
    epochs=EPOCHS_A, 
) 
 
graph_history(history) 
 
# Phase B 
model.get_layer('efficient_net_v2b0_backbone').trainable = True 
model.summary() 
 
history2 = model.fit( 
    train_ds, 
    validation_data=validation_ds, 
    epochs=EPOCHS_B, 
) 
 
graph_history(history2)
				
			

Notice that results will be different each run, if at some point you want to save your trained model just do:

model.save(MODEL_FILE_NAME)

And to load it:

model = keras.models.load_model(MODEL_FILE_NAME)

A possible (Phase A) graph: 

Results & final words 

Training graphs are great, but in the end, there’s nothing like seeing real examples along with the network prediction, so let’s have our model predict the validation split categories and show these. Let’s add some more code to achieve this: 

				
					results = [] 
classes = {0: "sun", 1: "moon"} 
 
# Accumulate all predictions 
for tensors in validation_ds: 
    images, labels = tensors[0], tensors[1] 
    predictions = model.predict(images) 
 
    for idx, image in enumerate(images): 
        results.append({ 
            'image': image, 
            'real_class': classes[tf.math.argmax(labels[idx]).numpy()], 
            'pred_class': classes[tf.math.argmax(predictions[idx]).numpy()] 
        }) 
 
# Show them, 4 at a time 
ic_fig = plt.figure(num=1, figsize=(8, 8)) 
ic_count = 0 
 
for entry in results: 
    sp = ic_fig.add_subplot(2, 2, ic_count + 1) 
    sp.set_title(entry['real_class'] + ' / ' + entry['pred_class'], size=8) 
    plt.imshow(keras.utils.array_to_img(entry['image'])) 
    ic_count += 1 
 
    if ic_count == 4: 
        plt.show() 
        ic_fig = plt.figure(num=1, figsize=(8, 8)) 
        ic_count = 0 
 
if ic_count > 0: 
    plt.show() 

				
			


We have certainly achieved a lot in this walkthrough, the following repository is available containing all the code we have reviewed so far: https://github.com/gweisz-folder/sun-moon-demo.git. 

 

Thanks 

Finally, I’d like to extend a big thanks to those who made the dataset possible by leveraging open pictures! 

Suns: 

Photo by CHUTTERSNAP on Unsplash 

Photo by Daoudi Aissa on Unsplash 

Photo by Dawid ZawiÅ‚a on Unsplash 

Photo by Ivan Torres on Unsplash 

Photo by James Day on Unsplash 

Photo by Jonathan Borba on Unsplash 

Photo by Nic Y-C on Unsplash 

Photo by Sara Kurfeß on Unsplash 

Photo by Selvan B on Unsplash 

Photo by Vivek Doshi on Unsplash 

Photo by Ishan @seefromthesky on Unsplash 

Photo by Jared Rice on Unsplash 

Photo by Jordan Wozniak on Unsplash 

Photo by Mourya Pranay on Unsplash 

Photo by Tschernjawski Sergej on Unsplash 

 

Moons: 

Photo by Altınay Dinç on Unsplash 

Photo by Bechir Kaddech on Unsplash 

Photo by CHUTTERSNAP on Unsplash 

Photo by Ganapathy Kumar on Unsplash 

Photo by Giovanni Pellizzari on Unsplash 

Photo by Griffin Wooldridge on Unsplash 

Photo by Guzmán Barquín on Unsplash 

Photo by Jordan Steranka on Unsplash 

Photo by LucasVphotos on Unsplash 

Photo by é‚± 严 on Unsplash 

Photo by malith d karunarathne on Unsplash 

Photo by Pedro Lastra on Unsplash 

Photo by sangam sharma on Unsplash 

Photo by Shot by Cerqueira on Unsplash 

Photo by Vino Li on Unsplash 

 

Sun & moon (Thanks page background): 

Photo by Jongsun Lee on Unsplash

Why you should augment your team with Folder IT

Outsourcing or Augmenting your IT Team with Folder IT professionals is a cost effective solution that does not sacrifice on quality nor communication effectiveness. Our teams are qualified for working with all the latest technologies and for joining you right away.

Request a quote now for outsourcing your project or staff augmentation services to Argentina. 

Tags

Access top talent now!

Related

Get in Touch