You have my sympathies, Furball, as I have been through this, and so have many of my friends, including my fellow cofounders of IFWG Publishing. This was part of the reason why we founded our company.
I have read some revealing articles by publishers and agents about what motivates acceptance of new talent etc, and it is a tangled web – very difficult to separate and thoroughly analyze. I think we all know that one telling factor is simply the skills and taste (and perhaps even the moods) of agents and submission editors. They are busy folk and they rely on a trained eye and their "gut feel" when skimming query letters and synopses (if they get as far as synopses). Many will claim that they are so well trained they can tell by the quality of the query letter whether it is worth steaming on or not, but there are plenty of authors who have been rejected, to only "come good" by the 100th attempt, to question the quality of these folk who represented the 99 initial rejections. I wish there was a database out there somewhere that keeps track of agencies that missed opportunities. It would be telling indeed.
Then there is simple business. Most large publishing houses (and reflected by the backdrop of agents) make most of their money from established writers or celebrities. This is where they spend most of their marketing money and they know that newbies don’t return much with their first few Titles. They are not very investment oriented – they want the fast buck, often driven by their shareholders. This doesn’t help the new writer trying to get a break.
There are other reasons, but it is pointless to go on. You know, despite this, I genuinely believe that a good writer will come good eventually, more often than not. I also believe that even established writers have to market themselves – it is a simple fact, and there is good literature out there on that topic as well. As one wise editor once said, "once you get published, that is when the journey begins." So guys, those of you who haven’t been published, remember there is a journey after THAT.
Some of the strategies that I think can help get the foot in the door are related to credentials and publicity. And which tool is most important to get there? Your writing. Try to win competitions. Try to sell as many short stories to magazines and anthologies (print and electronic) as you can. Target the bigger names – for instance, there are a bunch of sci-fi magazines where if you publish with them, you automatically qualify for entry into the Science Fiction Writers of America association. Every milestone will be another dot point on your CV which is attached to your query letter. Another way to get cred is to publish through companies like ours – and you have to help work the publicity. Once you sell enough units, you get street cred. People sit up and take notice. There are notable examples of this happening (think John Grisham, think Christopher Paolini, think Matthew Reilly).
Sorry, Furball – started to soapbox. In a nutshell, you are dead right. I say don’t give up, though – build a network of friends in the industry and place yourself in a position where luck is minimized.