So, in my version of Budapest, they were already partners and it was just another mission, not particularly hard one, but it did go sideways. One of the bad guys shot Clint with something that looked like tranquilizer shot, only it caused him hallucinations. Oops. So "You and I remember Budapest differently" translates into you were shooting things and I was stoned. The very line in the film functions as something that lets us know these two have shared history, and while it is a running joke (on tumblr for example), it never really bugged me, I always thought it's a hint in the direction of having history together, in fact enough history to make jokes while shooting at ugly aliens in the middle of New York.