In the study of grammar, the verb 'to modify' has a specific and limited technical meaning. It is used to mean the effect an adverb has on a verb. Given that verbs denote actions, and states, and adverbs 'describe' them, 'to modify' has been chosen as the word that accounts precisely for the relationship between verb and adverb, as it derives from the Latin modus, meaning 'way' or 'manner'. In other words, an adverb tells us more about 'the way [the action of] a verb is performed, or done'. It is not quite as simple as that: few things in grammar are. But it may help a beginning student to remember the accepted usage.
In traditional grammar as taught in UK schools in the first half of the twentieth century and beyond, the distinction between adjectives qualifying nouns and adverbs modifying verbs was held to be of great importance. As OED gives meaning 5 a. of modify as "Grammar. To limit or qualify the sense of (a word, phrase, or sentence); to stand as a modifier in relation to (a noun)", and one (subordinate) meaning of qualify as "Of an adv.: To modify", I cannot now find it of huge significance - unless you are writing for a pedant.