Many businesses don't understand to ask this. What does your agency outsource? If they outsource a majority of their work overseas, the quality of the content, design, and even language could suffer. Find out where your content is coming from up front or you may be disappointed later.
If you can't get a hold of your firm or it takes weeks to get changes made, they're not doing you much good. A firm that is available to you is your best friend. Now, that doesn't mean that there is any agency that can offer you 24/7 availability, but a quick response and a firm committed to customer service will make all the difference.
Many businesses look at marketing as a black hole because they have never dealt with an agency (or internal team) that has been able to produce and prove a positive ROI. Find out what kind of reporting they will offer, what success will look like, and how they'll help you set your goals.
If the agency that you're thinking about working with doesn't follow any of the guidance that they're providing you, that should be a red flag. Sure, it's easy for any agency to get caught up in marketing for their clients and do less of it for themselves, but would you really hire a company to build your website if theirs is terrible? Make sure they're drinking the same thing they're serving you (at least to some extent.)
You will probably have multiple people working on and collaborating on your account, but it's important to have one specific point of contact -- your Account Manager. This person will be able to communicate to the different departments on all of your projects and keep you updated with time lines, proofs, content, ideas, and more. Making sure you meet and are comfortable with this person can be a make or break relationship.
For most businesses, hiring a marketing agency is a great move. It will typically be cheaper than hiring an internal person (or team), you get a team of knowledgeable professionals, don't have to provide benefits, pay payroll taxes, and you don't have to worry about workman's comp or unemployment insurance. It is, however, important to do your homework on the agency that you decide to hire, ask good questions, and make sure you feel comfortable with them.
Before hiring an agency, you should make sure that you know exactly how much you have budgeted for marketing (and each form of marketing).