As always I start with saying that is a decision each should take based on the project and needs. The web is full of disputes and recommendations, based on what is trending or is cooler. Here is how I see it, in plain text.
So, starting with a bit of history ... and at the beginning was the pixel. Well, yes, I remember having to draw things on screen with Basic (scripting language for the new ones in the branch) so each pixel was defined based on its x-y coordinates. Thankfully you don't have to do this anymore, the software will do this for you.
PX
We learned that the pixel is the smallest unit you have on a display, it was used for a long time also to define fixed sizes for fonts or containers, before the responsive layout concept. If you need a certain container to keep a constant size you just say how many pixels it should have. The same goes for the texts. You can't have half a pixel, although now is a trend to fake that too using one of the 3 LEDs from a physical pixel. I would not go into that.
REM
When we thought that we mastered the sizes with percents and pixels we got hit by a new trend ... rem. And no, is not the musical band REM, is another unit. A unit made to scale. But how? Well ... rem is depending on the root font size. For now, the root is still the html tag. By default (also coming from ancient computing, I will not keep history lessons today) the html font size or the root font size is 16px, if we do not define it otherwise. So 1rem = 16px. Cool!! We can get fancy now and write 0.75rem instead of 12px. If we do some things and modify the root at runtime we will modify also the rem. For example if we define html font size as 12px, 1rem will be now 12px so 0.75rem will be now just 9px.
EM
When we thought that all is clear we got the trend of em units. Basically is an rem but based on the container in which is used. Or, to make it more hairy relative to the font-size of its direct or nearest parent, due to the inheritance properties of HTML.
One thing to note is that if we do not define any font size at a container level, then 1rem = 1em = 16px, if we define just the root 1rem = 1em = root size in pixels.
If we define a container having the font size 18px, then 2em will be 36px regardless of the root font size, and only in this container. And if in this container we define another one but with the font size of 10px, for this inner container 2em = 20px.
px, rem or em?
First of all you need to keep in mind that the browser will translate all the units in pixels at render time. Depending on your case worth to have different units or not.
If you don't need to change sizes
One downside I see is that many times the result will be something like x.zyw pixels which will be rounded anyways by the browser - up or down to the nearest pixel it can fill on the screen, so if you have three containers with the computed width of 20.857px you can end up having two of them with 20px and one with 21px, or two with 21px and one with 20px - depending on the width of the parent container. Although 1px is not visible, when you have a lot of things which add in this way you can start to see small misalignment.
Another downside is that you will always have to transform from px to rem or em, and always keep in mind the parent font size if is defined and how. Especially if you use em for paddings or margins, you will have to define the font size of the element, even if it will have no text, or the child container will have a text but with a different font size. From a pragmatic approach if you do not need scalable units is a waste of time to compute something just to be reverted by the browser when rendered, not to mention the errors coming from the human factor.
I see no use and waste of time, and I would rather stick to the fixed pixels than to be trendy with rem or em or both.
If you need to change sizes
On the other hand, if you manipulate the font sizes at runtime then is much easier to change just the root font size and based on this all rem units will change, rather than modifying all the values one after another. Also if you want to have a certain container where the padding or whatever sizes are depending on the container font size or its parent font size, then bring the em at the party. Then, you need to modify just the root and everything magically changes keeping their ratio.
Here you will need to think clear what exactly you expect and go for one method. I personally tend to skip em units if I need a constant unit for padding, margins, font sizes. It can happen that I need to change also the font size and then if I would work with em I would need to compute again how many em is 10px if now my font is 12px and not 14px. So it would require more attention at the inheritance.
In my work I have never needed actually the use of em, although some are saying this is the new black. I have not needed a padding depending on a certain font size so bad that I could not do it with rem.