6 Success Tips From the Man Who Built a $150 Million Speaking Agency Bernie Swain had no experience in his field when he quit his job and started the Washington Speakers Bureau in 1980.

Three U.S. presidents -- Ronald Reagan, George H. W. Bush and George W. Bush -- four prime ministers of Great Britain -- Margaret Thatcher, John Major, Tony Blair and Gordon Brown -- and other leaders including Arianna Huffington, Steve Case and Tom Brokaw.

These are just a few clients of the Washington Speakers Bureau. The agency, founded in 1980 by Bernie Swain, now boasts of $150 million in annual sales and reaches more than 2.6 million people a year.

And it all started on a whim.

At 35, Swain had already established a successful career as the assistant athletic director at George Washington University, where he had the chance to develop his skills in networking and building relationships. He would have been a shoo-in for the top athletic director position.

Related: Making the Leap of Faith to Entrepreneurship

"I threw it all away to take a risk on an entrepreneurial endeavor, which I had no experience in or a plan on how to succeed," Swain, now 70, tells Entrepreneur.

Success did not come easy.

"We sat there for a good 12 months, not representing anybody, not having a product," he says. "It was just a fluke that after 12 months, and months within going out of business, with no money, we found one speaker, one product, to build the Washington Speakers Bureau."

That first client was Steve Bell, a former Good Morning America anchor. The inexperienced Washington Speakers Bureau team signed Bell based on nothing more than a handshake. This turned out to be a blessing in disguise, Swain says.

Bell told others about the small upstart firm that would represent you without a contract, and new clients started rolling in.

"What makes what we ended up accomplishing so dramatic and amazing is we went from absolutely knowing nothing to reinventing ourselves and learning on our own without the advantage of anyone to teach us," Swain says. "We were simply making decisions day to day."

Here are some pieces of wisdom from Swain, whose book, What Made Me Who I Am, hits stores on Sept. 6.

1. On pursuing entrepreneurship

"When do you pursue entrepreneurship? When you have a passion that is so strong it overwhelms everything else you do in your life. We all have talent doing different things, but unless we have the passion to do it, we don't wake up every morning eager to overcome the difficulties and hard times that every entrepreneur faces when they go into business."

Related: 4 Tips From a 26-Year-Old Who Built a $10 Million Consulting Business

2. On why entrepreneurs fail

"The reason entrepreneurs fail today is they have talent doing something but not the passion to do it, so when things became too hard for them, they too easily give up. There are scores of entrepreneurial stories where people just hung on and made it work."

3. On effective networking

"The key for networking is not just letting people know who you are but building a relationship so they want to work with you. Networking equals relationships. 'Can I trust you?' 'Do you care about me?' It exposes the strengths and weaknesses of who you are and the company you're building. The important thing there is: Do things the right way. 'Am I making these decisions for the right reason?' 'Am I putting my customers first?'"

4. On managing big personalities

"I decided right from in the beginning, no matter who I represented and how famous people were, I had to establish a relationship that was real and honest. What they needed and wanted from me was to tell them the truth. If something didn't work, I had to say it. In return, I had to listen to what they had to say. I think that's the reason we ended up representing so many famous people on a handshake.

"I've never represented anyone I didn't love or have respect for. I showed respect to them, and in turn they respected me. I never had a problem with the people I represented. I wish they were my next-door neighbors."

Related: 10 Timeless Tips for Giving Effective Presentations

5. On being a good leader

"I wanted to get into business on my own because I wanted to make my own decisions. Inherently, that could mean I could be a dictator and you're going to have to live with those decisions. But part of being a good leader is developing a moral standard, establishing goals and allowing people to make their own decisions."

6. On what it takes to be a good public speaker

"The thing about the people I represent is that they know their subjects. The key is to know the subject matter. The second thing is, it is difficult to give a speech. You have to practice that speech over and over in front of people. You've got to practice it as well as any musician practices an instrument."