Seeking Advice on Designing Engaging Boss Battles for a Mario Fangame

[DJ Coco]

Boss fights are fun when you are constantly engaged in the fight.
Try to avoid waiting games where the player needs to wait for their turn to attack. A good example for a boss fight are the ones in Super Mario Odyssey - most boss attacks can be interrupted by the player if they are skilled enough, which allows speedrunners to take them out very quickly.

EDIT: To answer your questions

What makes a boss battle memorable for you in a Mario game?

A great boss fight has an interesting attack pattern, is satisfying to deal damage to, and delivers on the spectacle scale as well. Sometimes people use "perceived danger" in boss fights, which is something that looks intimidating and scary, but usually has little way of hurting the player. This helps make even simpler and easier fights feel memorable and cool. (For instance, adding lava to a bottomless pit does not change how the fight plays out, but it might feel more climactic.)

Are there any Mario fangames that have particularly well-designed bosses?

Unfortunately it's been so long I played fangames that my memory is hazy at best, but I remember Psycho Waluigi was quite creative with its fights. In my own fangame "Paper Mario 3D Land", Chaoxys had helped me create the boss fights, and I think the most well-designed one might be "Loco Poco" from World 2. Instead of waiting for an opening in the boss's attack pattern, the player needs to hit three POW blocks. If the player has the mole power-up equipped the player can deal double damage to quickly take him out once he's vulnerable.

What are some common mistakes to avoid when designing a boss fight?

Common mistakes include having hitboxes that are too big and poorly telegraphed attacks, which might lead to the player taking hits that they feel are unfair.
For instance, if a boss attacks with a sword, there should be a wind up animation, so the player knows "Oh, now I should get out of the way!". It's not fun if the boss starts an attack out of nowhere, and it's similarly not fun if the player feels like he dodged the attack, but still took damage. Make sure hitboxes are always smaller than the sprite!

Any recommendations on balancing difficulty?

Try to have playtesters that tell you their thoughts! A general rule of thumb is that if it feels slightly too easy for you, the developer, it usually is just right.
As a developer we naturally get very good at our own game, and something that really tests our skills might be fun to us, but usually is too overwhelming for the player.

If you want to provide a real challenge for those who mastered the game, put it in an extra mode, but try to keep the difficulty moderately low for those who just want to continue through the story.