{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "5_Final.ipynb", "provenance": [], "collapsed_sections": [], "mount_file_id": "1dxn-q2qFyRPRaZQDjgsElWDXuJvfOTRE", "authorship_tag": "ABX9TyPqFpbqpzyGLfv9bpRkkDvG", "include_colab_link": true }, "kernelspec": { "name": "python3", "display_name": "Python 3" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "TwF2OsIADC3R" }, "source": [ "# Results\n", "| Sl No. | Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4\n", "| - | --------------------- | ----------- | -- | -- | -- |\n", "| 1. | Attention Model (greedy search) | 0.306819 | 0.302596 | 0.339031 | \t0.383689 |\n", "| 2. | Custom Final Model (greedy search) | 0.214501 |\t0.243265 |\t0.303785 |\t0.36675 |\n", "| 3. | Simple Encoder Decoder (greedy search) | 0.317412 |\t0.308454 |\t0.333496 |\t0.366244 |" ] }, { "cell_type": "markdown", "metadata": { "id": "SJ0sI6tsDCyN" }, "source": [ "These are the best models of each approach in terms of bleu score. Here the list is sorted based on **cumulative BLEU-4** score. The best model we obtained was Attention Model(greedy search). This model performed far better than other two models since it was able to output names of diseases and some observation more correctly than others. Other models only outputted the caption assuming all the datapoints shown were of \"no disease category\". The simple encoder decoder model was only able to output one caption for the entire dataset suggesting overfitting. The final model's grammar was entirely out of place suggesting underfitting. More data with much more variability specifically X-rays with diseases can be much more helpful for the models to understand better and can help reduce bias towards \"no disease category\". Now I will create a final pipeline function where it takes two images filepaths as input and outputs the predicted caption." ] }, { "cell_type": "code", "metadata": { "id": "BHAmaMcHwSnu" }, "source": [ "image_folder = '/content/drive/My Drive/Medical image Reporting/Images'\n", "folder_name = '/content/drive/MyDrive/Medical image Reporting/pickle_files'\n", "tr_file_name = 'train.pkl'\n", "te_file_name = 'test.pkl'\n", "chexnet_weights = '/content/drive/My Drive/Medical image Reporting/ChexNet weights/brucechou1983_CheXNet_Keras_0.3.0_weights.h5'" ], "execution_count": 2, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "CUhN0eYiJNXR" }, "source": [ "import warnings\n", "warnings.filterwarnings('ignore')\n", "import joblib\n", "import os\n", "import tensorflow as tf\n", "import tensorflow as tf\n", "from tensorflow.keras.layers import Dense,GlobalAveragePooling2D, Input, Embedding, LSTM,Dot,Reshape,Concatenate,BatchNormalization, GlobalMaxPooling2D, Dropout, Add, MaxPooling2D, GRU, AveragePooling2D\n", "from tensorflow.keras.preprocessing.text import Tokenizer\n", "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", "import pandas as pd\n", "import numpy as np\n", "import cv2\n", "from nltk.translate.bleu_score import sentence_bleu" ], "execution_count": 3, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "7UK3t_hmg0Pk" }, "source": [ "##Chexnet" ] }, { "cell_type": "code", "metadata": { "id": "EMulX3Cpb_iI" }, "source": [ "#chexnet weights ; https://drive.google.com/file/d/19BllaOvs2x5PLV_vlWMy4i8LapLb2j6b/view\n", "def create_chexnet(chexnet_weights = chexnet_weights,input_size=(224,224)):\n", " \"\"\"\n", " chexnet_weights: weights value in .h5 format of chexnet\n", " creates a chexnet model with preloaded weights present in chexnet_weights file\n", " \"\"\"\n", " model = tf.keras.applications.DenseNet121(include_top=False,input_shape = input_size+(3,)) #importing densenet the last layer will be a relu activation layer\n", "\n", " #we need to load the weights so setting the architecture of the model as same as the one of the chexnet\n", " x = model.output #output from chexnet\n", " x = GlobalAveragePooling2D()(x)\n", " x = Dense(14, activation=\"sigmoid\", name=\"chexnet_output\")(x) #here activation is sigmoid as seen in research paper\n", "\n", " chexnet = tf.keras.Model(inputs = model.input,outputs = x)\n", " chexnet.load_weights(chexnet_weights)\n", " chexnet = tf.keras.Model(inputs = model.input,outputs = chexnet.layers[-3].output) #we will be taking the 3rd last layer (here it is layer before global avgpooling)\n", " #since we are using attention here\n", " return chexnet" ], "execution_count": 4, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "fpjA5KLRg2JG" }, "source": [ "##Image Encoder layer" ] }, { "cell_type": "code", "metadata": { "id": "xFKBWgISgh--" }, "source": [ "class Image_encoder(tf.keras.layers.Layer):\n", " \"\"\"\n", " This layer will output image backbone features after passing it through chexnet\n", " \"\"\"\n", " def __init__(self,\n", " name = \"image_encoder_block\"\n", " ):\n", " super().__init__()\n", " self.chexnet = create_chexnet(input_size = (224,224))\n", " self.chexnet.trainable = False\n", " self.avgpool = AveragePooling2D()\n", " # for i in range(10): #the last 10 layers of chexnet will be trained\n", " # self.chexnet.layers[-i].trainable = True\n", " \n", " def call(self,data):\n", " op = self.chexnet(data) #op shape: (None,7,7,1024)\n", " op = self.avgpool(op) #op shape (None,3,3,1024)\n", " op = tf.reshape(op,shape = (-1,op.shape[1]*op.shape[2],op.shape[3])) #op shape: (None,9,1024)\n", " return op" ], "execution_count": 5, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "2MDQovEqg6M6" }, "source": [ "##Encoder" ] }, { "cell_type": "code", "metadata": { "id": "DWB0ySadgv_g" }, "source": [ "def encoder(image1,image2,dense_dim,dropout_rate):\n", " \"\"\"\n", " Takes image1,image2\n", " gets the final encoded vector of these\n", " \"\"\"\n", " #image1\n", " im_encoder = Image_encoder()\n", " bkfeat1 = im_encoder(image1) #shape: (None,9,1024)\n", " bk_dense = Dense(dense_dim,name = 'bkdense',activation = 'relu') #shape: (None,9,512)\n", " bkfeat1 = bk_dense(bkfeat1)\n", "\n", " #image2\n", " bkfeat2 = im_encoder(image2) #shape: (None,9,1024)\n", " bkfeat2 = bk_dense(bkfeat2) #shape: (None,9,512)\n", "\n", "\n", " #combining image1 and image2\n", " concat = Concatenate(axis=1)([bkfeat1,bkfeat2]) #concatenating through the second axis shape: (None,18,1024)\n", " bn = BatchNormalization(name = \"encoder_batch_norm\")(concat) \n", " dropout = Dropout(dropout_rate,name = \"encoder_dropout\")(bn)\n", " return dropout\n" ], "execution_count": 6, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "1PBrs7PyhA-E" }, "source": [ "## Global Attention Layer" ] }, { "cell_type": "code", "metadata": { "id": "Tc9OVU5Kg-NZ" }, "source": [ "class global_attention(tf.keras.layers.Layer):\n", " \"\"\"\n", " calculate global attention\n", " \"\"\"\n", " def __init__(self,dense_dim):\n", " super().__init__()\n", " # Intialize variables needed for Concat score function here\n", " self.W1 = Dense(units = dense_dim) #weight matrix of shape enc_units*dense_dim\n", " self.W2 = Dense(units = dense_dim) #weight matrix of shape dec_units*dense_dim\n", " self.V = Dense(units = 1) #weight matrix of shape dense_dim*1 \n", " #op (None,98,1)\n", "\n", "\n", " def call(self,encoder_output,decoder_h): #here the encoded output will be the concatted image bk features shape: (None,98,dense_dim)\n", " decoder_h = tf.expand_dims(decoder_h,axis=1) #shape: (None,1,dense_dim)\n", " tanh_input = self.W1(encoder_output) + self.W2(decoder_h) #ouput_shape: batch_size*98*dense_dim\n", " tanh_output = tf.nn.tanh(tanh_input)\n", " attention_weights = tf.nn.softmax(self.V(tanh_output),axis=1) #shape= batch_size*98*1 getting attention alphas\n", " op = attention_weights*encoder_output#op_shape: batch_size*98*dense_dim multiply all aplhas with corresponding context vector\n", " context_vector = tf.reduce_sum(op,axis=1) #summing all context vector over the time period ie input length, output_shape: batch_size*dense_dim\n", "\n", "\n", " return context_vector,attention_weights" ], "execution_count": 7, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "FBp48aThhMdw" }, "source": [ "## One Step Decoder" ] }, { "cell_type": "code", "metadata": { "id": "GE2zcndehOxF" }, "source": [ "class One_Step_Decoder(tf.keras.layers.Layer):\n", " \"\"\"\n", " decodes a single token\n", " \"\"\"\n", " def __init__(self,vocab_size, embedding_dim, max_pad, dense_dim ,name = \"onestepdecoder\"):\n", " # Initialize decoder embedding layer, LSTM and any other objects needed\n", " super().__init__()\n", " self.dense_dim = dense_dim\n", " self.embedding = Embedding(input_dim = vocab_size+1,\n", " output_dim = embedding_dim,\n", " input_length=max_pad,\n", " mask_zero=True, \n", " name = 'onestepdecoder_embedding'\n", " )\n", " self.LSTM = GRU(units=self.dense_dim,\n", " # return_sequences=True,\n", " return_state=True,\n", " name = 'onestepdecoder_LSTM'\n", " )\n", " self.attention = global_attention(dense_dim = dense_dim)\n", " self.concat = Concatenate(axis=-1)\n", " self.dense = Dense(dense_dim,name = 'onestepdecoder_embedding_dense',activation = 'relu')\n", " self.final = Dense(vocab_size+1,activation='softmax')\n", " self.concat = Concatenate(axis=-1)\n", " self.add =Add()\n", " @tf.function\n", " def call(self,input_to_decoder, encoder_output, decoder_h):#,decoder_c):\n", " '''\n", " One step decoder mechanisim step by step:\n", " A. Pass the input_to_decoder to the embedding layer and then get the output(batch_size,1,embedding_dim)\n", " B. Using the encoder_output and decoder hidden state, compute the context vector.\n", " C. Concat the context vector with the step A output\n", " D. Pass the Step-C output to LSTM/GRU and get the decoder output and states(hidden and cell state)\n", " E. Pass the decoder output to dense layer(vocab size) and store the result into output.\n", " F. Return the states from step D, output from Step E, attention weights from Step -B\n", "\n", " here state_h,state_c are decoder states\n", " '''\n", " embedding_op = self.embedding(input_to_decoder) #output shape = batch_size*1*embedding_shape (only 1 token)\n", " \n", "\n", " context_vector,attention_weights = self.attention(encoder_output,decoder_h) #passing hidden state h of decoder and encoder output\n", " #context_vector shape: batch_size*dense_dim we need to add time dimension\n", " context_vector_time_axis = tf.expand_dims(context_vector,axis=1)\n", " #now we will combine attention output context vector with next word input to the lstm here we will be teacher forcing\n", " concat_input = self.concat([context_vector_time_axis,embedding_op])#output dimension = batch_size*input_length(here it is 1)*(dense_dim+embedding_dim)\n", " \n", " output,decoder_h = self.LSTM(concat_input,initial_state = decoder_h)\n", " #output shape = batch*1*dense_dim and decoder_h,decoder_c has shape = batch*dense_dim\n", " #we need to remove the time axis from this decoder_output\n", " \n", "\n", " output = self.final(output)#shape = batch_size*decoder vocab size\n", " return output,decoder_h,attention_weights" ], "execution_count": 8, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "W_ak-IiWhZp0" }, "source": [ "## Decoder Model" ] }, { "cell_type": "code", "metadata": { "id": "rSpwhAzXhbzR" }, "source": [ "class decoder(tf.keras.Model):\n", " \"\"\"\n", " Decodes the encoder output and caption\n", " \"\"\"\n", " def __init__(self,max_pad, embedding_dim,dense_dim,batch_size ,vocab_size):\n", " super().__init__()\n", " self.onestepdecoder = One_Step_Decoder(vocab_size = vocab_size, embedding_dim = embedding_dim, max_pad = max_pad, dense_dim = dense_dim)\n", " self.output_array = tf.TensorArray(tf.float32,size=max_pad)\n", " self.max_pad = max_pad\n", " self.batch_size = batch_size\n", " self.dense_dim =dense_dim\n", " \n", " @tf.function\n", " def call(self,encoder_output,caption):#,decoder_h,decoder_c): #caption : (None,max_pad), encoder_output: (None,dense_dim)\n", " decoder_h, decoder_c = tf.zeros_like(encoder_output[:,0]), tf.zeros_like(encoder_output[:,0]) #decoder_h, decoder_c\n", " output_array = tf.TensorArray(tf.float32,size=self.max_pad)\n", " for timestep in range(self.max_pad): #iterating through all timesteps ie through max_pad\n", " output,decoder_h,attention_weights = self.onestepdecoder(caption[:,timestep:timestep+1], encoder_output, decoder_h)\n", " output_array = output_array.write(timestep,output) #timestep*batch_size*vocab_size\n", "\n", " self.output_array = tf.transpose(output_array.stack(),[1,0,2]) #.stack :Return the values in the TensorArray as a stacked Tensor.)\n", " #shape output_array: (batch_size,max_pad,vocab_size)\n", " return self.output_array" ], "execution_count": 9, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "3DmqmApMIpKe" }, "source": [ "## Create Model" ] }, { "cell_type": "code", "metadata": { "id": "RWhc1AbHeeWt" }, "source": [ "def create_model():\n", " \"\"\"\n", " creates the best model ie the attention model\n", " and returns the model after loading the weights\n", " and also the tokenizer\n", " \"\"\"\n", " #hyperparameters\n", " input_size = (224,224)\n", " tokenizer = joblib.load('/content/drive/MyDrive/Medical image Reporting/tokenizer.pkl')\n", " max_pad = 29\n", " batch_size = 100\n", " vocab_size = len(tokenizer.word_index)\n", " embedding_dim = 300\n", " dense_dim = 512\n", " lstm_units = dense_dim\n", " dropout_rate = 0.2\n", "\n", "\n", " tf.keras.backend.clear_session()\n", " image1 = Input(shape = (input_size + (3,))) #shape = 224,224,3\n", " image2 = Input(shape = (input_size + (3,))) #https://www.w3resource.com/python-exercises/tuple/python-tuple-exercise-5.php\n", " caption = Input(shape = (max_pad,))\n", "\n", " encoder_output = encoder(image1,image2,dense_dim,dropout_rate) #shape: (None,28,512)\n", "\n", " output = decoder(max_pad, embedding_dim,dense_dim,batch_size ,vocab_size)(encoder_output,caption)\n", " model = tf.keras.Model(inputs = [image1,image2,caption], outputs = output)\n", " model_filename = 'Encoder_Decoder_global_attention.h5'\n", " model_save = os.path.join('/content/drive/My Drive/Medical image Reporting',model_filename)\n", " model.load_weights(model_save)\n", "\n", " return model,tokenizer" ], "execution_count": 10, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "xeUL835zIuB1" }, "source": [ "## Greedy Search Predict" ] }, { "cell_type": "code", "metadata": { "id": "rg4x_D6Mfysv" }, "source": [ "def greedy_search_predict(image1,image2,model,tokenizer,input_size = (224,224)):\n", " \"\"\"\n", " Given paths to two x-ray images predicts the impression part of the x-ray in a greedy search algorithm\n", " \"\"\"\n", " image1 = tf.expand_dims(cv2.resize(image1,input_size,interpolation = cv2.INTER_NEAREST),axis=0) #introduce batch and resize\n", " image2 = tf.expand_dims(cv2.resize(image2,input_size,interpolation = cv2.INTER_NEAREST),axis=0)\n", " image1 = model.get_layer('image_encoder')(image1)\n", " image2 = model.get_layer('image_encoder')(image2)\n", " image1 = model.get_layer('bkdense')(image1)\n", " image2 = model.get_layer('bkdense')(image2)\n", "\n", " concat = model.get_layer('concatenate')([image1,image2])\n", " enc_op = model.get_layer('encoder_batch_norm')(concat) \n", " enc_op = model.get_layer('encoder_dropout')(enc_op) #this is the output from encoder\n", "\n", "\n", " decoder_h,decoder_c = tf.zeros_like(enc_op[:,0]),tf.zeros_like(enc_op[:,0])\n", " a = []\n", " pred = []\n", " max_pad = 29\n", " for i in range(max_pad):\n", " if i==0: #if first word\n", " caption = np.array(tokenizer.texts_to_sequences([''])) #shape: (1,1)\n", " output,decoder_h,attention_weights = model.get_layer('decoder').onestepdecoder(caption,enc_op,decoder_h)#,decoder_c) decoder_c,\n", "\n", " #prediction\n", " max_prob = tf.argmax(output,axis=-1) #tf.Tensor of shape = (1,1)\n", " caption = np.array([max_prob]) #will be sent to onstepdecoder for next iteration\n", " if max_prob==np.squeeze(tokenizer.texts_to_sequences([''])): \n", " break;\n", " else:\n", " a.append(tf.squeeze(max_prob).numpy())\n", " return tokenizer.sequences_to_texts([a])[0] #here output would be 1,1 so subscripting to open the array" ], "execution_count": 11, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "QG0TOVh4IzT1" }, "source": [ "## Get Bleu" ] }, { "cell_type": "code", "metadata": { "id": "ql10gM9s7EiH" }, "source": [ "def get_bleu(reference,prediction):\n", " \"\"\"\n", " Given a reference and prediction string, outputs the 1-gram,2-gram,3-gram and 4-gram bleu scores\n", " \"\"\"\n", " reference = [reference.split()] #should be in an array (cos of multiple references can be there here only 1)\n", " prediction = prediction.split()\n", " bleu1 = sentence_bleu(reference,prediction,weights = (1,0,0,0))\n", " bleu2 = sentence_bleu(reference,prediction,weights = (0.5,0.5,0,0))\n", " bleu3 = sentence_bleu(reference,prediction,weights = (0.33,0.33,0.33,0))\n", " bleu4 = sentence_bleu(reference,prediction,weights = (0.25,0.25,0.25,0.25))\n", "\n", " return bleu1,bleu2,bleu3,bleu4" ], "execution_count": 12, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "lA4DovXGI1Ul" }, "source": [ "## Predict1 function" ] }, { "cell_type": "code", "metadata": { "id": "FXxFpnnJNmiA" }, "source": [ "def predict1(image1,image2=None,model_tokenizer = None):\n", " \"\"\"given image1 and image 2 filepaths returns the predicted caption,\n", " the model_tokenizer will contain stored model_weights and tokenizer \n", " \"\"\"\n", " if image2 == None: #if only 1 image file is given\n", " image2 = image1\n", "\n", " try:\n", " image1 = cv2.imread(image1,cv2.IMREAD_UNCHANGED)/255 \n", " image2 = cv2.imread(image2,cv2.IMREAD_UNCHANGED)/255\n", " except:\n", " return print(\"Must be an image\")\n", "\n", " if model_tokenizer == None:\n", " model,tokenizer = create_model()\n", " else:\n", " model,tokenizer = model_tokenizer[0],model_tokenizer[1]\n", " predicted_caption = greedy_search_predict(image1,image2,model,tokenizer)\n", "\n", " return predicted_caption" ], "execution_count": 13, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "b41EsGujI3ug" }, "source": [ "## Predict2 function" ] }, { "cell_type": "code", "metadata": { "id": "qSWfiq9Co_mU" }, "source": [ "def predict2(true_caption, image1,image2=None,model_tokenizer = None):\n", " \"\"\"given image1 and image 2 filepaths and the true_caption\n", " returns the mean of cumulative ngram bleu scores where n=1,2,3,4,\n", " the model_tokenizer will contain stored model_weights and tokenizer \n", " \"\"\"\n", " if image2 == None: #if only 1 image file is given\n", " image2 = image1\n", "\n", " try:\n", " image1 = cv2.imread(image1,cv2.IMREAD_UNCHANGED)/255 \n", " image2 = cv2.imread(image2,cv2.IMREAD_UNCHANGED)/255\n", " except:\n", " return print(\"Must be an image\")\n", "\n", " if model_tokenizer == None:\n", " model,tokenizer = create_model()\n", " else:\n", " model,tokenizer = model_tokenizer[0],model_tokenizer[1]\n", " predicted_caption = greedy_search_predict(image1,image2,model,tokenizer)\n", "\n", " _ = get_bleu(true_caption,predicted_caption)\n", " _ = list(_)\n", " return pd.DataFrame([_],columns = ['bleu1','bleu2','bleu3','bleu4'])" ], "execution_count": 14, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "E7En5-SPI_eX" }, "source": [ "# Function1" ] }, { "cell_type": "code", "metadata": { "id": "GPCgKPYRkwI8" }, "source": [ "def function1(image1,image2):\n", " \"\"\"\n", " here image1 and image2 will be a list of image\n", " filepaths and outputs the resulting captions as a list\n", " \"\"\"\n", " model_tokenizer = list(create_model())\n", " predicted_caption = []\n", " for i1,i2 in zip(image1,image2):\n", " caption = predict1(i1,i2,model_tokenizer)\n", " predicted_caption.append(caption)\n", "\n", " return predicted_caption" ], "execution_count": 15, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "60ft8xMBJCCZ" }, "source": [ "# Function2" ] }, { "cell_type": "code", "metadata": { "id": "zKc-JJ8qAyBh" }, "source": [ "def function2(true_caption,image1,image2):\n", " \"\"\"\n", " here true_caption,image1 and image2 will be a list of true_captions and image\n", " filepaths and outputs the resulting bleu_scores\n", " as a dataframe\n", " \"\"\"\n", " model_tokenizer = list(create_model())\n", " predicted = pd.DataFrame(columns = ['bleu1','bleu2','bleu3','bleu4'])\n", " for c,i1,i2 in zip(true_caption,image1,image2):\n", " caption = predict2(c,i1,i2,model_tokenizer)\n", " predicted = predicted.append(caption,ignore_index = True)\n", "\n", " return predicted" ], "execution_count": 16, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "V0Nea_Z913DI" }, "source": [ "df = pd.read_pickle(os.path.join(folder_name,tr_file_name))" ], "execution_count": 17, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "hMbEvITkeK9i", "outputId": "da6cd756-bf95-4b51-cd23-06620222323c" }, "source": [ "%%time\n", "k = [7,300]\n", "image1,image2 = df.iloc[k]['image_1'].values,df.iloc[k]['image_2'].values\n", "result = function1(image1,image2)\n", "print(result)\n", "print(\"len\",len(result))" ], "execution_count": 30, "outputs": [ { "output_type": "stream", "text": [ "['heart size is normal . lungs are clear . no nodules masses or adenopathy . calcified right paratracheal and right hilar lymph unchanged .', 'low lung volumes with no definite acute findings .']\n", "len 2\n", "CPU times: user 8.08 s, sys: 193 ms, total: 8.28 s\n", "Wall time: 7.75 s\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9l0AVBIniwRw", "outputId": "5e44c6b8-c2f0-4319-f8dc-f1610c7bbfe8" }, "source": [ "%%time\n", "k = [1,5,10,25]\n", "caption,image1,image2 = df.iloc[k]['impression'].values,df.iloc[k]['image_1'].values,df.iloc[k]['image_2'].values\n", "result = function2(caption,image1,image2)\n", "print(result)\n", "print(\"length of the result:\",len(result))" ], "execution_count": 29, "outputs": [ { "output_type": "stream", "text": [ " bleu1 bleu2 bleu3 bleu4\n", "0 0.8 0.632456 0.514316 0.604275\n", "1 0.6 0.387298 0.534700 0.622333\n", "2 1.0 1.000000 1.000000 1.000000\n", "3 0.8 0.632456 0.514316 0.604275\n", "length of the result: 4\n", "CPU times: user 9.9 s, sys: 314 ms, total: 10.2 s\n", "Wall time: 9.21 s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "UMW2HSFKzVhY" }, "source": [ "# Final Inference" ] }, { "cell_type": "markdown", "metadata": { "id": "AttSjCZxz8e_" }, "source": [ "## Long Sentence" ] }, { "cell_type": "code", "metadata": { "id": "0Y8Eghkv1bZd" }, "source": [ "test = pd.read_pickle(os.path.join(folder_name,te_file_name))" ], "execution_count": 20, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "USiOzx-857lF", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "ae8b181b-4891-4361-e6fe-50295b367c69" }, "source": [ "%%time\n", "k = 562\n", "caption,image1,image2 = [test['impression'][k]],[test['image_1'][k]],[test['image_2'][k]]\n", "predicted_caption = function1(image1,image2)\n", "print(\"true_caption:\\n\",caption[0])\n", "print(\"predicted_caption:\\n\",predicted_caption[0])\n", "print('bleu_score:\\n',function2(caption,image1,image2))" ], "execution_count": 26, "outputs": [ { "output_type": "stream", "text": [ "true_caption:\n", " large right pleural effusion with associated passive atelectasis of the right middle and lower lobes . grossly clear left lung .\n", "predicted_caption:\n", " partially loculated right pleural effusion grossly stable . stable moderate layering left pleural effusion . bibasilar airspace disease possibly atelectasis .\n", "bleu_score:\n", " bleu1 bleu2 bleu3 bleu4\n", "0 0.380952 0.19518 0.128736 0.211607\n", "CPU times: user 14.5 s, sys: 303 ms, total: 14.8 s\n", "Wall time: 14.3 s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "V-8N8AFEAH-h" }, "source": [ "We can see from the above prediction that the sentence created was grammarly ok suggesting the model is working well for long sentences. Comparing the true caption and predicted caption, we can see words like atelactasis,right and left pleural effusion being in the predicted caption suggesting the model is able to understand some of the diseases." ] }, { "cell_type": "markdown", "metadata": { "id": "HijuY-Py8JWu" }, "source": [ "## Medium Sentence" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6N8B5Mxl0QHG", "outputId": "ff0d0c80-39af-4766-ccf0-c4fb81ab92f1" }, "source": [ "%%time\n", "k = 541\n", "caption,image1,image2 = [test['impression'][k]],[test['image_1'][k]],[test['image_2'][k]]\n", "predicted_caption = function1(image1,image2)\n", "print(\"true_caption:\\n\",caption[0])\n", "print(\"predicted_caption:\\n\",predicted_caption[0])\n", "print('bleu_score:\\n',function2(caption,image1,image2))" ], "execution_count": 27, "outputs": [ { "output_type": "stream", "text": [ "true_caption:\n", " ['emphysematous change without acute radiographic cardiopulmonary process .']\n", "predicted_caption:\n", " no acute cardiopulmonary abnormality . chronic changes consistent with emphysema .\n", "bleu_score:\n", " bleu1 bleu2 bleu3 bleu4\n", "0 0.272727 0.522233 0.651314 0.722657\n", "CPU times: user 14.4 s, sys: 312 ms, total: 14.7 s\n", "Wall time: 14.2 s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "gZ0Ps2lmwRsr" }, "source": [ "We can see that the model is predicting the sentences that are linguistically correct. Grammar is also preserved. The model is working good for medium sentence." ] }, { "cell_type": "markdown", "metadata": { "id": "WyCP8ZUR8Llu" }, "source": [ "## Short sentence" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "tJBkhkmM-bx_", "outputId": "1bd1bd34-fa6a-4d70-b469-f878e25f10cc" }, "source": [ "%%time\n", "k = 451\n", "caption,image1,image2 = [test['impression'][k]],[test['image_1'][k]],[test['image_2'][k]]\n", "predicted_caption = function1(image1,image2)\n", "print(\"true_caption:\\n\",caption[0])\n", "print(\"predicted_caption:\\n\",predicted_caption[0])\n", "print('bleu_score:\\n',function2(caption,image1,image2))" ], "execution_count": 28, "outputs": [ { "output_type": "stream", "text": [ "true_caption:\n", " ['no acute cardiopulmonary abnormality . no evidence of active tuberculosis .']\n", "predicted_caption:\n", " no acute cardiopulmonary disease .\n", "bleu_score:\n", " bleu1 bleu2 bleu3 bleu4\n", "0 0.240955 0.190492 0.154909 0.182004\n", "CPU times: user 14.4 s, sys: 287 ms, total: 14.7 s\n", "Wall time: 14.3 s\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "-W7eoPcywnNC" }, "source": [ "The model has predicted first half of the true caption correctly but it didn't predict the other half but still meanings are the same." ] }, { "cell_type": "markdown", "metadata": { "id": "txcoeHQYw6FB" }, "source": [ "# Conclusion" ] }, { "cell_type": "markdown", "metadata": { "id": "grCXQ9qSw77q" }, "source": [ "We can see how the model is able to predict some of the captions correctly which have similiar meanings. Since the dataset given have a lot of datapoints that have similiar captions in meaning ie of one catgeory \"no acute cardiopulmonary abnormality\" or \"no active disease\", every model is showing bias towards this category even after applying upsampling and downsampling. This shows that we need a much larger dataset which not only include images of \"no disease\" category but also of images that have diseases will improve the model performance. The 3rd model will improve a lot if the dataset is large since some of the captions it produced were grammatically incorrect suggesting underfitting.\n", "\n", "The attention model (ie the best model) is predicting sentences that are grammatically correct even for long sentences." ] } ] }