[This was meant to be a contribution to a book collecting chapters on various topics that would help "a programmer who has never been politically active, but is starting to wonder why so many things seem to be going from bad to worse". Unfortunately the book has fallen through, so I'm posting the chapter here instead.]

What is a competitive marketplace

If you grow up in a capitalist society such as the United States of America, you end up being exposed to an idea called a "competitive marketplace". Such a thing is when you allow economic forces to dictate what the best solution to a problem is without the interference of other things (e.g. regulations, government protections, etc). Typically you see this in action when you go out to buy something, e.g. a light bulb. But a competitive marketplace can be used in other situations as well that might not be so obvious to people, e.g. private medical care. While most people don't argue that competitive marketplaces work well when purchasing goods, do they always make sense when applied to social "goods"?

Their benefits

To begin, we should explain what makes a competitive marketplace beneficial. We will use the idea of buying a light bulb as our illustrative example. Think about when you go to buy a new light bulb at the hardware store: do you have only a handful of choices for a light bulb, or many? Chances are you have many (a quick search online at a hardware store chain listed 671 products under their "light bulb" category). The reason this occurs in a competitive marketplace is because when you allow people to openly compete they often attempt to win over the customer by providing a product that meets their exact need. For example, the lamp you're buying a light bulb for will have a specific socket type that the bulb must match. The lamp may also have some sizing restrictions due to the lamp shade used. You may want to have an energy-efficient bulb. You might even want a bulb that emits a certain colour of light. The key point, though, is that by providing variety the competitive marketplace raises the chance that you will find a product that comes as close as possible to what you want. That means that ideally the competitive marketplace provides choice.

A competitive marketplace also helps provide choices that are cheap. When only economics is in play for deciding what to purchase, it allows price to play into the decision-making process. That will cause people to provide the cheapest product possible that still meets the needs of the consumer who wants to buy a product in order to stand out against competitors. Continuing with our light bulb example, you might find more than one light bulb that fits your needs perfectly, which then allows you to choose the cheapest light bulb (all things being equal in your final decision). This all means that ideally a competitive marketplace leads to cheap choices.

All of this helps lead to a product being made perpetually available. When a competitive marketplace allows for consumers to buy the products they want at a price they are comfortable with, there will typically always be someone willing to produce a good to meet that need. There may be some elasticity in terms of cost depending on how much demand there is for an actual product, but in general as long as the demand is high enough to allow for a profit in selling a good, someone will produce that good to fit the market. What this means is that ideally competitive marketplaces help guarantee that something remains perpetually available.

Their drawbacks

Having choice when making a purchase that is cheap and always available sounds great. So when do competitive marketplaces tend to fall short? There are few ways where things can go awry, mucking with the final outcome and results of a competitive marketplace. One such issue is in the difficulty of measuring the outcome. If you can't easily make sure that what you are buying will do what you want or verify post-purchase that everything is going as you expect it ends up ruining the marketplace as you are then working from imperfect information. And when you start having imperfect information then vendors are no longer competing purely on price but potentially on who deceives you the best.

Taking our light bulb example, let's say that you are going to have a plant growing underneath your lamp, so you want a bulb that emits a very specific colour of light to maximize the plant's growth. The problem is that unless you have a device that can accurately measure the colour of light you will have to wait and see if your plant grows as expected with a properly functioning bulb before you know whether you got what you expected from the light bulb. And since knowing if your plant is growing better or not is nearly impossible to know without another plant to compare against, that puts you in a bad position when you need to replace that light bulb again, e.g. do you trust that manufacturer again or do you go with another one? Since you can't easily measure if the bulb did what it was supposed to do you essentially have to guess as to whether the manufacturer told the truth or lied.

Along the lines of the difficulty in measuring outcomes is not taking positive externalities into account. In case you have never heard of that term, think of it as positive side-effects to something that you can't directly measure from the thing that you're measuring. The classic of a positive externality is a person with no children paying taxes to support public schools. While the person with no kids gets no direct benefit from children getting a free education, there is a positive externality to the childless taxpayer of an educated workforce, e.g. one of the children who was educated at a school that person paid taxes to support may one day become a doctor that saves the childless person's life.

Going along with our light bulb example, let's say you can't decide if you should pay extra for a brighter bulb or not. Up to this point the lumens that the broken bulb had seemed good enough. But what if it turns out that if you bought a brighter bulb you would read more because you didn't realize how much easier it is to read with a brighter light bulb? And what you read once led you to an idea that got you promoted at work? That would be a positive externality to getting a brighter light bulb that would not have been obvious while you were making your purchasing decision.

Since competitive marketplaces are driven by cost, a serious issue can be a misalignment of the economic incentives. This misalignment can happen both on the consumer or producer side of the equation. For the consumer it can come down to a simple misunderstanding of how one should tie cost to what they need. In our light bulb case this could stem from someone evaluating cost based on how long a bulb is slated to last without taking into consideration the failure rate of the light bulb and its warranty (e.g. a bulb might be slated to last 20 years, but if the warranty is only for a single year and after 2 years the bulbs have a 50% failure rate then the bulb should really only be expected to last 11 years on average which is well short of the expected 20 years). On the other side, producers can end up with the wrong incentives economically as well, e.g. bulb manufacturers may calculate that they can have a high failure rate after 5 years because people will forget how long they were promised the bulb would actually last, allowing the producer to make cheaper bulbs without any ramifications to their market share.

Probably the biggest issue with competitive marketplaces is that monopolies develop which prevent a pure competitive marketplace from existing. Ideally there shouldn't be monopolies so that the economic benefits of a competitive marketplace can occur, but for various reasons this rarely happens. Sometimes the formation of monopolies are due to a scarce resource and so a competitive marketplace led to a monopoly. For instance, let's assume that light bulbs need to use some rare-earth metal and there's only a handful of mines that can procure the metal. If a light bulb manufacturer bought all of those mines and didn't sell to competitors then owner of the mines would corner the light bulb market, allowing them to sell light bulbs at any price they choose.

Now one might argue that kind of abuse would either lead to alternatives being developed to fight the monopoly or that the government would step in (i.e. either market or governmental pressures, respectively). Unfortunately this doesn't always work out as there are ways to deal with this. For the market pressures, the manufacturer can either keep their prices just below the cusp of triggering a market revolt, or they can artificially create competition by allowing only a handful of competitors access to the scarce resource which allows for only restricted competition. As for governmental intervention, that assumes the government is entirely altruistic in how the regulations are put in place to stop the monopoly. There is also the potential issue that the rules to fight the monopoly might lead to other issues as there are now regulations which still prevent a competitive marketplace to exist that is driven entirely by economic forces. And since there is no marketplace that has absolutely zero regulations, there typically is no large-scale competitive marketplaces in the world.

Finally, the last issue is when a competitive marketplace is opened to procure something that is then given exclusive rights to the market. This often happens for high-cost items where the long-term costs are massive enough that it is considered better to sign an exclusivity contract with a producer. Taking our light bulb example, imagine that the United States government decided to sign an exclusive supplier contract with one light bulb manufacturer. While the competition may be open to all manufacturers for the exclusive contract, once the final contract is signed to get light bulbs at a discount, that is the only supplier that the US federal government will buy from for the next 10 years. This can be quite the problem if the light bulb manufacturer, knowing that the the US federal government has no choice but to buy bulbs from them at a set price, starts to cut corners to have a bigger margin. Now one might argue that problem is because the exclusivity contract was written poorly that something like that could happen, and that's true. But there's a problem that writing ironclad contracts is extremely difficult, and with economic incentives pushing producers to cut corners when they have no competitors then it typically leads to this sort of exclusivity arrangement not benefiting the consumer.

Example: private prisons

As you can see there's definitely an attractiveness to competitive marketplaces, but they are also very hard to make work. The attractiveness of competitive marketplaces has led to governments trying to use them for some services, but then falling into various traps that make the competitive marketplace they created not work. Take prisons as an example. Governments always need to have prisons to house those who have broken the law. Prisons are also not cheap, with costs covering initial construction, upkeep of the physical building, staffing, and costs associated with keeping prisoners housed and fed. Due to prisons having a high cost, some governments have attempted to set up competitive marketplaces to find the best deal they can for private companies to run the prisons they must have in perpetuity. Unfortunately these private prisons have historically not worked out well.

To begin with, prisons are not a good you can easily substitute out if you find you don't like your supplier. Once you set up a prison with a specific provider you can't change said provider on a whim as prisons can't have any downtime (you don't exactly want to have a prison with no guards for a day while you change vendors, for instance). This has led to governments having a competitive marketplace for the procurement of a private prison, but then once a provider is chosen they receive an exclusive contract. This runs afoul of the monopoly issue post-procurement which can lead to prison providers cutting corners once they have their exclusive contract.

Even more troubling is that there are major positive externalities that are hard to put a price on that allow for a proper bidding process in the competitive marketplace. In the case of prisons, you want a prisoner to not re-offend once released. Unfortunately preventing people from re-offending does not directly tie into the day-to-day cost of running a prison cheaply. In fact one could argue that trying to rehabilitate prisoners runs counter to running a prison cheaply as that would mean job training, counselors, etc. which all cost money. If you are paid to simply run a prison then those things which help minimize re-offence are superfluous costs, while from the government's and general populace's perspective it's an extremely important positive externality to minimize the chances someone convicted of a crime will re-offend.

While it may be possible to write a contract with a private prison that ties re-offending to some economic punishment for the prison provider, economic incentives may not line up with that desire in the end. For instance, if a government signs a contract with a prison provider that states that inmates should be taken care of such that they re-offend at a rate below say 40%, then the prison is economically motivated to get the recidivism rate to 39%. Unfortunately the prison provider is not incentivised economically to aim for below 39%. That means that while the government and general populace would want to see the prison provider aim for a 0% recidivism rate (which is unobtainable, but something to strive for), the prison provider will never aim for that due to the fact that it would cost them extra money which would cut down on their profits. In other words, in a gross perversion of societal needs versus economics, it's in the best interests of the prison provider from an economic perspective for prisoners to re-offend which is not what's best for society.

Example: private medical insurance

Another example of where competitive marketplaces don't necessarily work out well is private medical insurance. In case you're not from a country like the United States that doesn't have socialized medicine, private medical insurance has two costs to them. The first part is the premium, which is what you pay every month to have the medical insurance. The second part is the deductible, which sets the maximum you have to pay out of your own pocket to cover expenses. For example, if you have a $200/month premium with a $2,000 deductible that means you pay $200/month to have your medical insurance and you have to also cover the first $2,000 each year in medical expenses you incur (passed your $2,000 deductible you don't have to pay anything, although this is simplifying things greatly for illustrative purposes). The assumption is that by having a competitive marketplace for private medical insurance you can drive overall prices down for everyone.

The problem is that when it comes to a person's health, there are a lot of positive externalities in making sure the entire populace is healthy. When someone is ill, that directly impacts their family and their employer. It can also impact you if the sick person happens to provide a service to you that is impacted due to them missing work. There's also the possibility that they will get you sick. And finally, if the illness is serious enough to lead to hospitalization or death, those are very expensive situations to be in due to even more lost time at work, the cost of hospitals, and the impact on lives when someone passes unexpectedly. So in general it's considered a good thing to keep everyone as healthy as possible.

Unfortunately private medical insurance in a competitive marketplace does not promote this. To start with, it's really in the best interest of an insurer to keep you healthy enough to not go passed your annual deductible. As long as you don't start incurring costs on the insurer they have no economic incentive to keep you as healthy as possible, just healthy enough to not be a cost burden.

Then there is the issue of people who have pre-existing medical problems or are projected to become ill (e.g. the elderly or someone with a family history of an expensive illness). It is entirely in the interest of a private medical insurer to not cover someone who has known illnesses that will cost more than the person's annual deductible. That means either the insurer will simply not cover the person or require them to have such a high deductible (or high premiums to have a lower deductible) that it becomes cost-prohibitive for an individual to have health insurance. This then leads to people who are unlucky enough to have the biggest chance of missing work due to illness, becoming hospitalized if they are not properly taken care of, etc to actually not receive proper medical care as they cannot afford medical insurance. That then leads to them either impacting society monetarily by more hospital visits and indirectly by missing work, etc.

This is why nearly all countries that can afford public health coverage provide it universally. By prioritizing helping everyone, including those who have expensive upfront costs for their medical conditions, the overall system can work towards minimizing social impact from missing work but also from a health issue reaching the point of needing hospitalization which is much more expensive than taking care of the illness upfront. Public health coverage also allows the system to prioritize optimal health instead of the cheapest health for someone, which essentially leads to positive externalities regarding health.

Conclusion

Competitive marketplaces have their place. They help drive costs down, lead to better solutions to problems, and help prevent the permanent loss of a product or service from the market. But competitive marketplaces are also not the solution to every problem that has a cost associated with it. They do not do well when there is no flexibility in changing suppliers or positive externalities cannot be properly thought of ahead of time or accurately taken into consideration in calculating the cost of a solution. In other words, competitive marketplaces are great when you need to buy a pencil, but they are not so good when your well-being is at stake.