Hcfyk

$$text{If} cases{a+b+c=1 \ a^2+b^2+c^2=2 \a^3+b^3+c^3=3}
text{then} a^5+b^5+c^5= ?$$

A YouTuber solved this problem recently and, though he spent some time explaining it, took over 40 minutes to solve it.

Like the video, the best I can do with this is relying on expansion formulas and substitution. As trivial a problem this is, the numerous trinomials and binomials with mixed terms makes it very, very tedious.

What is the quickest/shortest approach to this problem (meaning it doesn't need to be solved algebraically)? You don't have to type the entire solution out, I think if I'm given a good hint then I can take it from there.

Let's start with the basic symmetric expressions: $ab+bc+ca$ and $abc$. You can refer to giannispapav's answer for details, which shows that
$$ab+bc+ca = -1/2, abc = 1/6.$$

With that, Vieta's formulas implies that $a,b,c$ satisfy:
$$ x^3 -x^2 - x/2 -1/6=0,tag{1}$$
Or
$$x^3 = x^2 + x/2 + 1/6.$$

That means, for $x$ equals $a,b,c$,
$$x^4 = x^3 + x^2/2 + x/6,$$
and
$$x^5 = x^4 + x^3/2 + x^2/6.$$
Adding the two equations above, we have
$$x^5 = frac32x^3 + frac23x^2 + frac16x.$$
Now replace $x$ as $a,b,c$ and add them all up, we have
$$a^5+b^5+c^5 = frac32(a^3+b^3+c^3) + frac23(a^2+b^2+c^2) + frac16(a+b+c).$$

Note: if you feels that
$$a^3+b^3+c^3 - 3abc = (a+b+c)(a^2+b^2+c^2 - ab - bc - ca)$$
is too complicated to verify, then Vieta's formulas is the way to go. That is, replace $a,b, c$ in Equation $1$ and add them up, where $1/6$ is indeed $abc$ as in Vieta's formula.

Using Newton's identities

$$
begin{aligned}
e_{1}&=p_{1}\
2e_{2}&=e_{1}p_{1}-p_{2}\
3e_{3}&=e_{2}p_{1}-e_{1}p_{2}+p_{3}\
4e_{4}&=e_{3}p_{1}-e_{2}p_{2}+e_{1}p_{3}-p_{4}\
5e_{5}&=e_{4}p_{1}-e_{3}p_{2}+e_{2}p_{3}-e_{1}p_{4}+p_{5}\
end{aligned}
$$
with $p_1=1,p_2=2,p_3=3,e_4=0, e_5=0$, we get $p_5 = 6$.

This answer is almost in the same spirit as @Quang Hoang's, but I hope this answer will add something. Let $$
P(z) = (z-a)(z-b)(z-c)=z^3-sigma_1 z^2+sigma_2 z-sigma_3
$$ where $sigma_1=a+b+c$, $sigma_2=ab+bc+ca$ and $sigma_3=abc$ by Vieta's formula. Note that for $zin {a,b,c}$,
$$
z^{n+3} =sigma_1 z^{n+2}-sigma_2 z^{n+1}+sigma_3 z^n,
$$ hence by summing over $zin {a,b,c}$, we get recurrence relation
$$
s_{n+3}= sigma_1 s_{n+2}-sigma_2 s_{n+1}+sigma_3 s_n
$$ for $s_n = a^n+b^n+c^n$. Given the data, it can be easily noted that $$sigma_1=1 ,quad sigma_2 =frac 12 left((a+b+c)^2-(a^2+b^2+c^2)right)=-frac 12.$$ And by plugging $n=0$, we obtain
$$
3=1cdot 2+frac 12cdot 1 +sigma_3 s_0=2.5 + sigma_3s_0,
$$ so $sigma_3=abcne 0$ and $s_0=a^0+b^0+c^0=3$. This gives $sigma_3=frac 1 6$, implying that
$$
s_{n+3}=s_{n+2}+frac 12 s_{n+1}+frac 1 6 s_{n},quad forall nge 0.
$$ Now $s_4 =frac {25}{6}$ and $s_5=6$ follows from the initial data $(s_3,s_2,s_1)=(3,2,1)$.

Note : The theory of homogeneous linear difference equations is behind it.

You can use

$a^4+b^4+c^4=(a^2+b^2+c^2)^2-2(a^2b^2+a^2c^2+b^2c^2)$,

$(ab+ac+bc)^2=a^2+b^2+c^2+2(ab+ac+bc)$,

$a^3+b^3+c^3-3abc=(a+b+c)(a^2+b^2+c^2-ab-ac-bc)$,

$(a+b+c)^5-a^5-b^5-c^5=(a+b)(a+c)(b+c)(a^2+b^2+c^2+ab+ac+bc)$

Fun video!

Much time was spent on finding $abc=1/6$.

Alternative method for this:
$$begin{align}a^2+b^2&=2-c^2 Rightarrow \
(a+b)^2-2ab&=2-c^2 Rightarrow \
(1-c)^2-2ab&=2-c^2 Rightarrow \
ab&=c^2-c-frac12 Rightarrow \
abc&=c^3-c^2-frac c2 end{align}$$
Similarly:
$$abc=a^3-a^2-frac a2\
abc=b^3-b^2-frac b2$$
Now adding them up:
$$3abc=(a^3+b^3+c^3)-(a^2+b^2+c^2)-frac12(a+b+c)=3-2-frac12 Rightarrow abc=frac16.$$
In fact, you can find other terms as well:
$$ab+bc+ca=(a^2+b^2+c^2)-(a+b+c)-frac32=2-1-frac32=-frac12;\
a^2b^2+b^2c^2+c^2a^2=ab(c^2-c-frac12)+bc(a^2-a-frac12)+ca(b^2-b-frac12)=\
abc(a+b+c)-3abc-frac12(ab+bc+ca)=\
frac16-frac12+frac14=-frac1{12}$$
Hence:
$$a^5+b^5+c^5=(a^2+b^2+c^2)(a^3+b^3+c^3)-(a^2b^2+b^2c^2+c^2a^2)+abc(ab+bc+ca)=\
2cdot 3-(-frac1{12})+frac16cdot (-frac12)=6.$$