Even after the Danville, Ky., left Kentucky in 2000, he still followed the Wildcat program. As Brown raised a young family and his career took him all over the country, the idea of returning to his roots flickered in the back of his mind.
This offseason, it became a reality.
"You always think about coming back but you never get your hopes up too much in this business because it's crazy and it's unpredictable," Brown said. "It just so happened that the stars aligned last December and it all worked out and I'm thankful for it."
As Brown suggests, he wasn't going to drop everything at the first opportunity to come home. As one of the promising young minds in college football, Brown had his future to think about. In order to put on UK blue again, the former wide receiver would have to be convinced.
To complicate matters, coming to Lexington to serve as offensive coordinator wasn't Brown's only opportunity this winter. Multiple schools courted him and some even thought of making the 33-year-old one of the youngest head coaches in the country.
Mark Stoops, meanwhile, stood aside. He knew he wanted Brown and he made that clear, but Stoops gave him time to sort through process. In the end, Stoops' quiet patience was his masterful final sales pitch. It showed Brown that Stoops only wants the best for those around him.
" 'This is a good guy. This is a good person. This is a guy that understands what we're doing on a day-to-day basis.' " Brown recalled telling his wife. "I said, 'I can work for him. I want to work for him.' "
With complete faith in Stoops' vision for the UK program, Brown signed on as the second assistant on the new UK staff on Dec. 10. Over the three days that followed, two more former Cats joined him.
First was John Schlarman. The former All-Southeastern Conference UK offensive lineman signed on to coach Wildcats at his former position and had feelings similar to Brown's about coming back to the Bluegrass and being close to extended family in his native Ft. Thomas, Ky.
"It's something that you hope for, you keep your fingers crossed someday you will have that chance but you really never know if it's going to come or not," Schlarman said.
"For me, I'm from Plant City, Fla., and I came 1,500-1,600 miles away to start my career," Scott said. "It's like coming back home to be honest with you. I really almost feel like I'm going back to my house. I know where I'm going, I know how to get around."
His wife, the former Shambrica Jones, played basketball at UK. Excited about being so close to her beloved UK Hoops program, she took over all the logistics of the move while Scott hit the recruiting trail.
"When I got this opportunity, my wife said, 'Just go, I'll handle everything,' " Scott said. "That's what I did. I came here and she stayed back I was here for about two and a half months before she came."
Unlike Scott and his wife, Brown, Schlarman and Scott weren't classmates at UK, but their stories intertwined.
Schlarman was first to arrive. He was a four-year starter from 1994-97 and Brown - a lifelong UK fan - can recall watching him play in both high school and college.
"I think it's a good lesson for our guys and if you just work hard and continue to get better at your craft, you can be an All-SEC player," Brown said.
As he always hoped, Brown came to Kentucky as well, starting his career the year after Schlarman graduated. In Brown's final season, 2000, Schlarman joined the UK staff as a graduate assistant. Watching Brown establish himself on the team, Schlarman was reminded of himself.
"He came in and worked hard, nothing was given to him and he had to earn every opportunity that he got while he was here," Schlarman said. "I was a little along those same lines."
Scott, meanwhile, arrived in 2000 as a gifted back, rushing for 611 yards and earning All-Southeastern Conference Freshman Team honors.
"We knew he was very talented from the word go," Schlarman said. "He was really fast back then and he's probably really fast still but I remember him being an explosive and dynamic running back. He went out there and worked hard and did some really good things very early in his career."
"When he came to us at Troy he was coming in as the guy that had been a graduate assistant in college and was a high school coach," Brown said of Schlarman. "Because he was a part of some really good offensive units, I think he's a more confident coach now."
In Schlarman's mind, that goes for Brown and Scott too. Watching them now, Schlarman still sees the players he remembers working with as a graduate assistant. But now, all three have become fully formed coaches.
"I think they always have a little bit of the same characteristics that they carried as players but as a coach you have to kind of get more vocal and that sort of thing," Schlarman said. "I think all of us have evolved in that regard. You have to coach and get your ideas across to those guys and not just worry about our little area."
Schlarman was separated from Brown and Scott from 2010 when they left for Texas Tech, but they are together again in a place they know well. When they were first hired by Stoops, fans and media talked extensively about the value of having three coaches so intimately familiar with the school on board.
Having been on the job, the three offensive assistants agree.
"I think just being familiar with everybody always helps," Schlarman said. "There's not that get-to-know-you period quite as much."
Brown, Scott and Schlarman have known their surroundings from day one. They know their way around campus and they see familiar faces daily, but that doesn't mean everything feels the same.
Though they all saw success as players and were members of some of the best UK teams in recent memory, the mood around the program is unmistakably different. They still haven't coached in a game yet and there's untold amounts of work yet to be done, but they all agree it feels like something big is happening.
"There was excitement around the program when I played, but the excitement around this program now is setting new heights," Scott said. "It's unbelievable."
The success the new staff has already had on the recruiting trail is certainly a part of that. So too is the passage of House Bill 7 that approved UK's pursuit of Commonwealth Stadium renovations. But when picking out a single highlight from his time on the job so far, Brown didn't hesitate in passing over both.
The Blue/White Spring Game is the easy choice.
An estimated 50,831 were in attendance for the annual scrimmage, giving UK the sixth-highest spring-game attendance in the NCAA in 2013. Looking around at the hundreds who came out early for the Cat Walk and the thousands who packed Commonwealth for the actual game, Brown saw a fan base taking ownership of its role in rebuilding the UK program. Nothing could be more exciting than that.
"I think it's a huge thing that the fans need to understand," Brown said. "Our fan support plays a huge role in the success we are going to have on the field and the success we have in recruiting."
From his childhood as a fan to his playing days and even to his time coaching at other schools, Brown has always believed in Kentucky football. Events like the spring game are bringing a lot more people to his side.
"Guys want to be part of that," Brown said. "Guys want to go where there's excitement and energy and those things could be felt that night. That was a huge day for UK football and I think it will pay off in the future."
This story appeared first in The Official 2013 University of Kentucky Football Yearbook, which is available for $10 at Kentucky outlets like Kroger, Fan Outfitters and most bookstores right now. To purchase it online, click on this link.