This paper begins with the question. Is it possible that a machine can understand and evaluate the meaning of a text? Word2Vec is a neural network method of computing vector representations of words, which means the tool processes a text corpus and produces numeric vectors on the meanings of the words. This paper seeks to examine whether the automatic evaluation metrics, BLEUmodif and METEORmodif, enhanced by word2vec technology, are able to capture the meaning of translated texts better than their standard versions. To this end, first, five literary texts written in English were translated into Korean by 119 students. Then, these translations were evaluated by two professional translators/teachers and by five different automatic evaluation programs. These are standard versions of BLEU and METEOR as well as BLEUmodif and METEORmodif with the word2vec technology and finally Sent2Vec that computes sentence vector representations. For the sake of convenience, the last three systems are referred to as “embedding versions” in this study. The analysis shows that overall, the standard version of METEOR performed the best (grades: 0.838, ranking: 0.727), but at the level of individual texts, the embedding versions of BLEU and METEOR showed a higher correlation with human evaluation (four texts) than their standard versions (one text).