Abirami - you should profile your code to see where the bottlenecks are. Visit the link profile for details on how to use this tool which will identify where your code spends the most time.
A couple of thoughts - I noticed that you have a two loops where you build the binary equivalent of the 256x256 image and the encryption key. Both of these loops assign the binary string to a cell array. It is good practice to allocate memory to the arrays before using them so that MATLAB does not have to resize them on each iteration (see pre-allocate for details). You could do something like
% pre-allocate memory for the 256x256 cell array
B = cell(256,256);
for i=1:1:256
for j=1:1:256
B{i,j} = dec2bin(A(i,j),8); %%conversion of pixel values to 8 bit binary
end
end
This may help, but maybe not as much as you would like. I think that the code is (initially) spending some time in the conversion of all 256*256=65536 (grayscale) pixels into their 8-bit string equivalents using dec2bin. I think that you can avoid all of these conversions (for the image and key) by just creating a map of all 256 integers to their 4-character DNA string. From
codebook1 = containers.Map({'00','11','10','01'},{'G','C','T','A'});
we can create a local map object that maps the values of 0-3 to each of the above characters
map = ['G';'A';'T';'C'];
Here we map '00' to index 1 which is 'G', '01' to index 2 which is 'A', etc. We can then build a 256x4 codebook that maps all possible integers from 0 to 255 (in indices 1 to 256) to the 4-character DNA codes as
codebook1a = char(zeros(256,4));
for k=0:255
x = dec2bin(k,8);
codebook1a(k+1,:) = [map(bin2dec(x(1:2))+1) map(bin2dec(x(3:4))+1) ...
map(bin2dec(x(5:6))+1) map(bin2dec(x(7:8))+1)];
end
Then create C which will be the 256x256 image converted from its 8-bit binary values to the 4-character DNA codes
C = arrayfun(@(x)codebook1a(x+1,:),A,'uni',0);
We add the one to account for the mapping of 0-255 to 1-256. This would replace the need to do any conversion from the 8-bit unsigned integer to a 8-character string to the 4-character DNA code.
We would do the same for the key as
O = arrayfun(@(x)codebook1a(x+1,:),M,'uni',0);
The above assumes that your input image is grayscale and each pixel is a 8-bit unsigned integer (which I think is a safe assumption given how the code has been constructed).
The above may only reduce the running time by a few seconds (with tic/toc I observed a 15 second improvement from the start of the program to the line where O is initialized).
After that comes the use of a function named chen1. This must be a custom function, so it is unclear what it does so perhaps the profiling will reveal something useful.
Definitely comment out the disp calls as they don't really add anything relevant - they just display the contents of the matrices in the Command Window. Consider using code similar to above to work the other way in the conversion of the DNA codes to the 8-bit unsigned integers. The function letterXOR maybe something else that you want to post as it must be another custom function that could be optimized.
