Yeah, it does sound like annoying behavior, but the kind that you guide kids for, not punish them. "Okay, being silly is fun, but that didn't sound funny to Grandpa. Can we think of some goofy things to say that are funny for everyone?" And then start cheerfully talking nonsense. Or say "It sounds as if you have lots of energy and need something to do with it," and suggest an activity that will wear them out a little bit.