You absolutely can do this, but there are two very important points to consider.

1. What is your purpose in choosing this ending?


2. In what way will this be a satisfying conclusion, from the reader's perspective?

In your question, you're describing a particular sequence of events as being an unexpected one. That's great, but what makes story isn't just the sequence of events, it's what those events mean.

There are lots of stories that end with the protagonist failed and crushed. Those are Tragedies, from Oedipus Rex to Hamlet to Watchmen. But these works all are tragedies long before their final scene -- the protagonist losing is the culmination of the tragedy; but its seeds have been planted and growing from the very start.

There are also many stories about how the universe is uncaring and capricious; how empires rise and fall based on bad luck and human perversity (Catch 22 comes to mind, as does A Series of Unfortunate Events). But that capriciousness doesn't begin in the last chapter -- it's baked in right from the start.

The reason I'm making these comparisons is because you're mostly describing this conclusion in terms of shock value. In terms of "being surprising."

But: A good story surprise isn't just "The reader wasn't expecting this." The reader doesn't expect the protagonist to suddenly die of a heart attack, or win the lottery, or be elected President by spontaneous write-in votes. It's easy for an author to do something the reader isn't expecting; the question is whether that twist works in service of a coherent story, or not.

That's why the two questions I opened with are so important:

1. 
   What is your purpose in choosing this ending? The ending shouldn't be shocking just for the sake of being unexpected. (Unless shocking twists are part of what makes your whole story interesting! In which case it's totally on-brand!) There should be some reason you consider this a good and interesting ending, besides that it's not what the reader was expecting.


2. 
   In what way will this be a satisfying conclusion, from the reader's perspective? The reader needs to feel the conclusion is satisfying. That it answers their questions; closes things off; fits with the rest. That, in some way or another, this is where the story was headed all along; that in retrospect, this is the fitting capstone for the entire story. You need to understand why this is true (or make it true, by making your story build up to it).

Knowing the answers to these questions will keep you focused on building towards your ending; on knowing how it's meant to work and how to accomplish that. That'll help address your questions on foreshadowing and effectiveness as well -- which depend heavily on what, exactly, you're trying to accomplish :)

Hero-always-wins is a trope

I wouldn't call this a plot twist. A twist is a reveal. It changes how events earlier in the story are perceived.

This is subverting a trope. The trope is an expected cliché: "the hero always wins", but then you break or subvert expectations. (See 2016 for middle-aged men having a cosmic meltdown because their Star Wars expectations were subverted.)

What you want to avoid is a shaggy dog ending: a long drawn out tale that has an abruptly anti-climactic ending that makes the rest of the story feel like a meaningless journey that went nowhere.

It also sounds like your story is (more or less) a tournament plot where the hero fights and wins, and then moves on to the next battle – I'm not suggesting that would be the only thing happening in your story, but as tournaments generally go one guy wins and one guy loses. It's probably not going to be that big of a surprise, unless the readers are really young and have only ever seen reassuring stories where the hero always unconditionally wins.

Always winning is a problem with any series. The formula meet, fight, win can be hard to break when the whole show is about an Awesomeguy™ who always wins. A one-off story doesn't suffer from this formula, but you may have an iconic character who doesn't really change. These characters can be harder to make interesting than a flawed character who needs to learn a few lessons and doesn't have such an easy path or assured victory.

Coming at the end, you are intending to trick the reader, not your protagonist. He can't learn from this defeat, it's the end of his story. If he's defeated at the beginning, or somewhere in the middle, he can change and grow. It can become his motivation (and fear). A defeat at the end isn't a lesson for this character, you're just trying to pull the rug out from under the reader. They might not appreciate being the butt of your joke.

What age is your reader?

Young children prefer repetitive stories that follow clear rules: heroes win, badguys lose, all is right in the world.

At some point (pre-puberty) this starts to shift and kids prefer a special individual to self-identify as the hero, often an underdog who levels up, or can change the outcome based on the force of her friendship charisma. Special individuals might be the chosen one or switched at birth – these characters have their own clichés and tropes to explain how they are simultaneously underdogs with superpowers.

With teens and young adults come the antihero, grim-dark, and villain-as-hero tropes. Here we see the protagonist isn't such a special individual, but the whole world is inherently unfair which justifies un-heroic behavior. The only reasonable character is a rebel. The only way to beat the game is to refuse to play. We don't expect these characters to win, but they might inflict some damage and get in a few one-liners. When they do win, it feels good but a little schmaltzy. The win is often unearned because the story flirted with a dark, no-win situation, but then retreated to the hero-always-wins tropes from kids stories. It's like eating frozen Tater-Tots® as an adult, you know it's not a meal but it recalls a lot of warm fuzzy childhood feelings.

But slowly as readers mature, they begin to enjoy characters who aren't necessarily special, navigating a world that is not particularly good or bad. As we grow-up we take responsibility for our own actions, we make our own life in the world, and we realize that interesting characters must make their own story happen, as opposed to just reacting to events that happen to them, or fulfilling a pre-defined destiny, or discovering they have won the genetic lottery and can defy physics with their brain. Mature readers enjoy believable characters in believable conflicts. You can still eat Tater-Tots for dinner but the nostalgia wears off, and there are so many other "mature tastes" to explore. Hero-always-wins isn't the expected trope anymore. It should be more complicated than that. Tater-Tots are only on the kids menu.

A hero represents our worldview

Little kids have no agency and depend on reliable caregivers to make the world more fair. Tweens have figured out that society rewards exceptionalism and privilege, so their heroes reflect this by being special from birth. Teens have some agency but not as much as they'd like, so they identify with rebels against an unfair world…. As we mature our tastes change. It's not a sudden cut-off, and we can still enjoy different kinds of stories, so this is a broad generalization to give you some things to think about.

Based on this (admittedly rough) guideline, I could say that your story is aimed at readers who are getting bored with the chosen one and hero-always-wins tropes, and might be pleasantly surprised by a grim-dark or villain-as-hero ending. You should consider how you can signal that this will turn out to be one of those stories. Don't "trick" the reader with a bait-and-switch. You don't want to fool little kids with a grim-dark ending, like Paddington Bear gets his skull smashed – that's just mean.

You should signal to the more mature readers that this won't be the "kids stuff" they are getting bored with. And you'll want to send the same signal to the younger ones who aren't ready to accept they are living in an unfair world, and being "special" doesn't always make up for it. If you deliberately lie to the reader about your story's worldview, it's a whole other issue than just subverting a trope or adding a plot twist. You are turning away the readers who will enjoy your story. Find ways to signal what kind of worldview your story takes place in. You can still entertain with surprises without resorting to a razor blade hidden in a Halloween apple.

You can absolutely do this

WARNING: This answer contains multiple spoilers, beware of things under the quotes

You aren't the first to think of this. Often this is used to setup a sequel but you can also subvert it. Below are a couple of well known examples, beware spoilers (you have been warned).

Title

Avengers: Infinity War

What happens

It's a super hero film, they are expected to win, that's basically how the genre works. Instead as "earths mightiest heroes" combine they unable to prevent Thanos from activating the Infinity Gantlet. Half the universe ceases to exist and the villain gets away.

Title

A Game of Thrones (Book)

What happens

The first book in the series sets Ned Stark up as the hero. He gets more narrative time than any other character, he is the most likable and noble of the bunch. This is the start of a heroes quest to route out corruption and bring the kingdoms into a new ages of prosperity. Until he gets brutally executed at the end of the book. No great salvation for this hero. He's dead.

So knowing that it isn't unheard of to do this, let's consider your questions.

Is this in general a poor way to end a story, or an effective one?

Would you judge either of those examples as a "poor story"? I certainly wouldn't. In fact I think the sudden plot twist has been a keystone of their success.

Would this kind of ending make sense, or would it be too sudden and immediately turn off the reader?

The key to avoid is your readers feeling cheated. Don't go out of your way to prove that the protagonist should win, they are already expecting that. Instead ensure that it is actually feasible that the antagonist could have won. Readers will hate it if your powerful hero is suddenly weak at the crucial moment. You need a realistic and believable way for your antagonist to have won.

Some readers may still be turned off. That's ok, your writing doesn't have to be for everyone. Others will love you for it. You are subverting a trope and if done well, on your way to a classic.

Should any instances of foreshadowing be implicated?

Foreshadow to your hearts content. Or don't foreshadow at all. It doesn't really make a difference. Readers will still believe the protagonist is going to win. That's what protagonists do.

How you use foreshadowing and set up the narrative is really beyond the scope of this question. Suffice to say that even if you straight say the antagonist is going to win, if readers like your protagonist they will still have hope that there will be a miraculous save in the end.

Would it best go down as a surprise moment?

To me, the surprise is the best part of the whole thing. So many novels are predicable to the point of being dull. I can often guess how the hero wins and which of their allies makes a noble sacrifice several chapters before the climax of a story. Breaking those expectations in refreshing, intriguing and brilliant writing.

You will get backlash

Killing off your protagonist will never be without risk. All authors face backlash when they get rid of a favourite character. Be prepared for this. That doesn't mean you shouldn't do it though. It means you got enough investment in your book that readers are having emotional responses to the words on the page. Isn't that the goal for any author?

Conclusion

You can and should do this if that is the story you want to write. Beware the backlash from devastated fans. But don't use that as an excuse not to write a great book.

where the antagonist wins by luck, so to speak.

When you say "wins by luck", do you mean a deus ex machina scenario? Such as, "the unstoppable hero is struck by lightning and dies, the villain wins due to an event outside of the control of the parties"? That would be very unsatisfying, not because the villain wins, but because I'd get the impression that the author wrote himself in a corner with an overpowered, invincible main character, and couldn't think of a better way to resolve the conflict than killing him by accident.

So I am wondering, is this in general a poor way to end a story, or an effective one? Would this kind of ending make sense, or would it be too sudden and immediately turn off the reader?

Having the villain win is not the issue. It's who wins, and subsequently, how they win, that is the crux of the matter.

If you want the villain to win at the end, for this twist to be satisfying, you should have a villain that's as interesting as the hero. Don't make him pure evil for the sake of being evil, make sure that the audience respects, empathizes, or is even charmed by the villain. If the villain wins, that makes them the new "protagonist", so to speak, so that they should be worthy of that role.

Another thing you can consider is the chemistry between the hero and the villain (see Batman vs the Joker), if that was developed in the prior chapters, it will add a layer of depth to both characters in their face-off.

Another thing you can do is make the final fight escalate, and ultimately turn into a Pyrrhic victory for the villain. The hero eventually loses, but they put up a good fight, and while they don't defeat the villain, they leave it permanently impaired in some way(s), so that "nobody wins", they both lose.

In any case, you should not rob the protagonist of a heroic death, they shouldn't lose to a stroke of bad luck. This time, especially the moment they realize that they're going to die, they should fight to the best of their ability, so that they can have a solemn death. If the villain has respect for the fallen hero, the audience will also have respect for the villain.

On the other end of the spectrum, and depending on the personality of your characters, their backstory, and your audience, in order to make the villain more likeable, you could make the hero turn villainous on the brink of their death. When I mentioned having a Pyrrhic victory before, it was implied that the hero would cause debilitating injuries in a fair fight. But, imagine if the hero uses an underhanded tactic to dishonorably maim the villain with their last breath, or, even worse, hurt someone dear to the villain.

Of course, this dark awakening would need a justification, it can't happen out of the blue: maybe the hero had used dirty tricks before, or maybe the hero feels like that they have no honor to lose if they employ cheap tactics now that they've been already defeated, or maybe the villain is so evil, that the hero feels like it's justifiable to give up their moral code, in one last desperate attempt.

It can definitely be done well. But there's a big pitfall you have to watch out for. You need to have some thematic reason for the protagonist to lose. Some character fault or thematic point you're making that causes his defeat. He can't simply lose. If there's no meaningful reason for his defeat, if he is just simply outmatched, your audience will wonder what the point was.

To combat this, you should have the character make mistakes due to his flaws or the theme of the story that set up his failure. Maybe he breaks a promise he thought was unimportant and the person he slighted betrays him. Maybe he's selfish and alienates his allies. Maybe he underestimates just how low the villain is willing to stoop to win.

I think one problem with this is as an ending is that it’s a very common plot twist—in the middle of a story. If you stop there, it doesn’t feel like something the reader’s never seen. It feels like a three-act story stopped at the end of Act I. One recent movie that reminded me of was Star Wars: The Force Awakens. Everyone knew it was the first of a trilogy, so

Many reviewers were surprised that Rey won against Kylo Ren at the end. I remember several asking how he could be a credible villain after that. There was also a backlash against the main character for being a wish-fulfillment character who always succeeded at everything, especially without needing to earn it.

The story structure that reminds me of the most, though, is sports movies. Those often have the structure that the main character needs to win a series of elimination matches to reach the championship, and there are some famous examples where the scrappy underdog narrowly loses, while winning everyone’s respect. You’re telling the story of the undefeated favorite who unexpectedly loses on a last-minute fluke. That is, the rival in a movie like The Karate Kid.

That’s not a terrible idea—there are, in fact, sequels from the perspective of the boy who fought The Karate Kid, and where the boxer who fought Rocky retires and becomes a mentor to his former rival after he too suffers shocking personal and athletic losses, and the original hero later mentors his rival-turned-mentor’s son. The so-called villains turn out to have been good sports with interesting stories of their own to tell.

A lot depends on execution. But. If you told a superhero story and end the first time the hero loses, or you tell an underdog sports story where we don’t get to know or root for the underdog, it would probably come across as incomplete.

There are a lot of interesting things you could do with that plot twist. Maybe the point is that life isn’t fair and we all need to learn to cope with adversity. Maybe it’s that the people we compete against are not less deserving or our enemies. Or that everybody’s luck runs out. All these themes are at least as old as Gilgamesh. Or maybe it’s just that the hero should train better and come back next season.

I think, if you worked out what the purpose of the twist ending is meant to be, you’d end up developing it and rewriting the story around it, so that the reversal of Fortune happens earlier and there is more time to react to it, or the reasons it happened are foreshadowed.

As a reader, my biggest problem with what you're describing isn't that the protagonist loses, but that he just loses at the last minute through plain bad luck. If you pull something like that out of nowhere, your readers will rightly feel cheated. You've invalidated their reason for spending however many hours reading what went before, because nothing before that point is relevant to the conclusion.

A common trope in bad writing is the deus ex machina, where an external element comes from nowhere and fixes everything. This is infamous in crime novels, of course, but it's not at all uncommon elsewhere. Your suggested story is also a deus ex machina ending, except with a malign deity, I suppose. Any writing resource can give you multiple reasons why a deus ex machina is lazy writing which makes your story a failure, so I won't list them all here.

If there's a strong reason for events to turn out unexpectedly like this - Ned Stark's death in Game of Thrones being a great example - then of course it is acceptable. But then you wouldn't be posting here, would you, because these events (however unlucky) would be key elements of your story, and you wouldn't be asking for this kind of advice. I think it's clear that in spite of the well-meaning advice from other answers, this concept cannot be saved. By all means have this outcome from the fight if you want, but you'll need to alter everything that went before to ensure that outcome is believable.

The idea that a hero always wins has been used too much and i think anyone who actually tries to break out of this mold is a rather creative and non -stereotypical writer, i.e an awesome writer!. But( a big one) you have to really know how to make the reader feel after the event. After such an event, the reader can feel one of these

1. Sympathy for the protagonist and anger for the antagonist.


2. A feeling of defeat if the reader had made up a picture of the protagonist as a competitor to the antagonist. Here the reader does not feel sympathy, he feels the same way someone feels when their favourite team loses to a team, and the loss was their fault.


3. A mixed feeling, or rather an confusion, where the reader tries to justify what was the right thing, which is, judging if it is the antagonist's win or the protagonist's loss. A good example of this is the movie Se7en. Which i will not spoil, but would rather recommend to you to watch.

Now that the reader side is over, we may move on to the antagonist. When a antagonist is defeating the protagonist, then his side of the story should also be justified i.e he should not be a mere flat character, or a sudden introduction at a part of the story. He can be

1. Built up at the same pace as that of the protagonist and made a much deeper character, leading to the reader justifying his actions 'deeply'. This would not potentially put off the reader even if he wins.


2. Be ambiguous but not invisible. And by that, i mean the antagonist should always be there in the story, but there would be no direct mention, he could be an overarching (maybe powerful) character who is always indirectly influencing the protagonist, and thus the reader. An example can be Handsome Jack in the game borderlands 2, although the protagonist finally wins.


3. This one is rather AWESOME if you can apply it. Introduce the antagonist in a very later part of the story, keep him ambiguous and at the end, make him win. Donot give any lead to justify your decision, or the win of the antagonist, but make the antagonist's small part in the story a very interesting and curious one, such that the reader is left asking questions about him. We may call the ending a cliffhanger as so much information is missing to justify it. And this cliffhanger will drive your story and the reader forward.
   But but but , that will not be the end of the story. You will include the first chapter of another book you will write, and that first chapter will just start the introduction of the antagonist. The reader will not only absolutely love your book but he will be wanting the next book too, because he just wants his answers to his questions. And thus you can or rather should write the story of the antagonist differently but involve the same ending. This way, the reader will be the one to decide who is the real pro/antagonist. A small example would be Russian roulette, which the tells the story of the assassin who tried to kill alex rider, the protagonist of the series of story books, alex rider by Anthony Horowitz.

I hope this helps you and as always happy writing!!

You can definitely subvert expectations like this and still create a great story. However, there's other issues with this story structure that should be kept in mind.

You write:

My protagonist is a skilled character whose powerful traits allow him
to succeed continuously throughout the story. He is challenged by foes
along the way, but he eventually overcomes them all.

I'd like to suggest that you might need to rethink your story just a little bit to avoid "concept-imprisonment", in which having a clear concept around which your writing twirls ultimately imprisons you as a writer, limits what you can do, and finally results in the creation of a one-dimensional "concept piece" that isn't as rich or enjoyable as you had originally hoped.

The danger here is really that the reader gets a sense of your protagonist's skill and ability too early on in the piece. You could easily end up with a story where there's really no tension, no adversity and no character development. This would be pretty boring, but in light of what you've described it's a real risk.

To keep yourself free enough to avoid falling into these traps, I recommend creating some get-out-of-jail free cards for yourself so that anytime you want to take the story in direction A but your grand concept plan suggests direction B, it remains possible to take direction A by playing one of these get-out-of-jail-free cards.

For example, in Thor Ragnarok, the almighty Thor has a chip in his neck for most of the movie that allows people carry the appropriate remote to cause him to become paralyzed and lose any fight he's about to win. This creates a lot of tension and uncertainty, and allows the writers to have him lose fights where the narrative demands it, resulting in more freedom for them and ultimately a better story. You could have something similar.

I also think that your story should be a trilogy. In the first book, you subvert expectations with the hero's unexpected loss. In the second and third books, he/she has to live with the consequences of this loss, learn some difficult lessons, and ultimately maybe they will triumph. In this way, you can subvert expectations while still crafting a story that leaves people feeling like a satisfying conclusion was reached.

Also, I think you should keep in mind that subverting expectations, while nice, only makes your story slightly better. Indeed, when the subverted expectations don't feel satisfying and don't play a role in a larger, more interesting narrative, they can feel pretty empty. With this in mind, I'd like to express disagreement with one of the comments made by another answerer.

This is subverting a trope. The trope is an expected cliché: "the hero
always wins", but then you break or subvert expectations. (See 2016
for middle-aged men having a cosmic meltdown because their Star Wars
expectations were subverted.) - wetcircuit

In my opinion, what let TLJ (The Last Jedi) down was not subverted expectations; indeed, this was one of the few strengths of the movie. What let TLJ down (aside from the annoying gender politics that burdened it down in so many ways) was that all the writer's creative energy was invested in subverting expectations, leaving not a lot of creative energy for other very important things. In particular (spoilers ahead):

There was a lack of attention to characterization and character development. Luke acted totally out of character the whole movie and the tiny amount of explanation we were given for his intense apathy and cynicism wasn't enough to make it feel like the same character. Yoda acted totally out of character and came off more like a book-burning nutjob than a wise elder. Rey's character was okay, but did not really develop in a meaningful way, and her power levels were way too high for no reason. Fin's character retrograded to make it seem like the last half hour of Force Awakens never happened, e.g. when he's trying to escape near the start of the movie. Poe's character was cool as always, but he already seemed fully developed before the movie began and the feeble attempts at creating human-growth opportunities felt forced... you know, I have this image of some really stern unpleasant woman trying to explain to Mohammad Ali that he's doing this whole life thing wrong, just several days before his monumental victory. This doesn't seem like an opportunity for growth. It just seems lame, and the audience naturally stops paying attention because it's just too stupid.

There was also a lack of attention to the canon. For example, Leia uses the force to levitate her way out of empty space and back into the safety of the ship. If the force can do this, why didn't Palpatine use this to save his arse at the end of Episode 6? As another example, Holdo weaponizes hyperdrive to devastating effect in TLJ, which was visually cool. But it does make you think about that A-wing that slammed into the executioner in Episode 6 - shouldn't that pilot have jumped to light-speed to increase the damage? Indeed, if hyperspeed is such a devastating weapon, wouldn't they have built missiles based on the technology long ago? Etc. etc.

It seems to me that most of the above problems probably arose because the director was so focused on surprising the audience that many other important story elements ended up taking a back seat.

So the lesson here is really, yes, do surprise your readers if you can, but only if it makes retrospective sense and fits into a bigger and grander story. Don't expect that just doing something unexpected will instantly make your story good. Interesting characters with relatable goals, sympathetic viewpoints, and the kinds of character flaws that real people have will do more for your story than cheaply subverting the reader's expectations all the time, and you need to keep in mind the different elements that go into a good story to get the balance right.

Above all, I just want to emphasize: don't write yourself into a corner where a single concept ends up reducing your story to a mere idea, or mere philosophy. The weakness of writing is that it's not empirical, it doesn't take into account statistics. You read one story at a time, and who knows how likely that story really is? But the strength is that you can explore details of existence, and really get involved in the messiness of life in a way that science cannot. And, you should play to this strength, because assuming you're writing for adults, exploring the messiness of life is going to be a key element of just about any great story.