Just playing your game, minding your business. Suddenly, the game crashes. Down in smoke. All unsaved progress lost. Sadness comes upon you. Your day is filled with despair. All your friends and family feel the weight of your grief. You decide to take it out on your simulated people. You let them jump in the pool. You take away the pool ladder. The ladder is the only way they know how to get in or out of the water. The ledge is much too slippery. Your simulated people perish in the swimming pool.

How quickly do our bugs escalate and get out of hand. What started off as an off-by-one error ended up in an of-by-a-million error because those ones keep adding up. That's just the problem with errors. Ignore one and things start to get out of hand. That's why you deal with them as they happen. How should you handle errors then? It all just depends on what your program is doing. Sometimes, an error is transient. For example, you try to connect to a web service, and it didn't work. Just try again and then it works. The same could be for reading a file. Try it again after a few moments, there you have it. File is read.

Consider the case where there is a transient error in some operation that your program is trying to perform, but that transient error is very common. Maybe it's something your program does once a minute with retries, but it has a 50% failure rate. If that happens, it can manifest itself as a performance problem in your program. If you didn't log anything or issue any warnings, your transient error could go unnoticed. What makes this kind of error so bad is that the retries make everything stable and working. Since everything is working, nobody cares to turn on the extra logging which would be required to see the transient errors. The transient errors, not being seen, go unresolved. From the point of view of the user, the program is just slow. The user inevitably gets frustrated by the performance of the program. Sooner or later, the simulated people are the ones who end up with the user's wrath. That, or, the user eventually switches to another program which doesn't have your performance issues. After all, no one likes to deal with a program that runs too slow. Too slow programs are going to be put back aside in feature of newer, faster programs. Yet still, the newer, faster programs could also have bugs that result in transient errors, and the cycle repeats. We hold endless funerals for our simulated people. Then we'll have to pull up a mausoleum simulator because of all the simulated people who meet the brunt end of our frustrations.

Log data can be a double edged sword, because logging is not free. Yes, you get lots of information about what the program is doing, but then you have the added overhead of the logging itself. That could potentially be problematic if you need your program to be as performant as possible. Another thing to consider is that sometimes extra logs might not actually add any additional troubleshooting or debugging value. For example, if you have a method that performs some calculation, whose result is deterministic according to its input, there might not be much value in adding logging to it. As long as you know what the inputs are, you can easily determine what the output would be. You don't want to prematurely optimize, but at the same time, there's no point in expending extra effort in writing logging code that doesn't give you any value, and may even be a hindrance in reading the log data or in the additional logging overhead.

What could be a good balance between excessive logging and making sure that transient errors stand out? Transient errors that are too common or have known causes you might hide from your logs, and just let them show up when you turn the logging level up. If you turn the logging level up, however, you might get extra log data that you just don't need during normal runs. There could be some kind of middle ground here. Maybe instead of making the logging level a fixed value, make it something that adjusts dynamically based on the conditions. For example, if there is performance degradation detected, maybe push up that logging level for a while. If an uncommon error occurs, that could also push up the logging level. There could be something like, a base logging level which is the one that is used by default, then there could be an "effective" logging level, which is one that starts at the base but then can adjust dynamically based on the conditions under which the program runs.

Transient errors might be hidden in the normal operation of the program, but then if things start getting slow, the dynamic logging level gets increased, and the transient errors that were hidden before start showing up in your logs. Then you have more visibility into issues like that without having to take steps at reconfiguring the program to get it to output additional logs.

Another aspect to consider is how to report these errors. No matter what logging level you use, or what trickery you might have in place to get transient errors exposed to you so you can do something about them, if no one reads your log messages, they still won't do any good. Telling the user about the log messages might not do much good either. You could do something like display a message to the user about these kind of logs, but users might not be able to do much about them or understand what they mean. You could upload error reports, but there's some privacy issues there, potentially. Error reports could contain sensitive information.