Image quality assessment without a reference image is essential for evaluating the performance of image enhancements. Much research has been done to develop an objective image quality measurement that is relevant to perceived quality evaluations. Because the contrast of an image is one of the important factors for a user to evaluate the image quality, methods for improving the contrast of images are also being extensively studied, but the assessment algorithms for evaluating them are limited. In this paper, we propose a contrast measurement of a color image based on a stimulus in the human visual system (HVS). The proposed method evaluates the luminance component of the image based on just-noticeable-difference to reflect the local contrast perceived in the HVS. The contrast in the color component is evaluated based on a model of the stimulus of the color component in the primary visual cortex (V1). The region response factor, which reflects the relative change of the color saturation in the different luminance values in V1, is used to image contrast. We tested the validity of the proposed method using various image databases and subjective tests. The experimental results showed that the proposed method had higher correlation with the evaluation of people than conventional methods of measuring image contrast without the original image.