As long as I'm not naked and without coin I'll continue trying.
What usually happens; 5 to nearly 10 tries later, people start to drop out or change out between toons and it all starts to consume a lot of time. Disconnect, porting issues, casting time, travel time and ofc the multiple of afk breaks to blow of some of that frustrationtsteam (usually I end up not using that afk break and continue brainstorming). Now if the group still happens to have the patience, we'll continue after regroup attempt X. If other players are involved, the attempts are far less and after another couple of whipes, the group just calls it.
Depending on what guild I'm in, I'll safe the area and call in guildmembers to focus down and work the strats (especially if it's a first kill boss). I'll start asking other good reputation known high end players about similar encounter or even how they've killed it or what to watch for.
In the scenario that I'm the one not able to meet the requirements, I'll bow out after I or the group has found a decent replacement.
I'll keep giving everything I've got until someone figures out how to defeat this boss or someone decides it's time to move on. I consider myself very patient guy, so I haven't expirienced a situation where I would initiate droping an endeavour in group setting (happend couple of times while I did something on my own, when I decided that I need to gather my strength and try again later).
Hold a grudge against the boss, level up, and destroy it later. And when I get frustrated at something months later, I use this once-difficult boss as my punching bag to relieve said stress.
Fippy hates me now.