I have flown an aircraft before and after prop balancing. It was done on aircraft dynamically. It made a noticeable difference, but this was on a PT6 so it is obviously running a lot smoother than any piston engine runs.
I would say its worth doing.
Last year we had a stone chip on the back of one blade of a Hartzell composite prop. It would have been nothing on a metal prop, but it cost around £3000 to repair. We don’t go to stony strips but we obviously found one somewhere.
I think you would feel it.
There is a long history of dynamic prop balancing and it seems that it can make a big difference if done right.
You can affect the balance of a prop on a simple balance stand by spraying a little more paint on one end of the prop. Whether that level of minor imbalance can be felt in service on the plane I don’t know.
Saw this in a magazine a few years ago but difficult now to find proof: post 6 on backcountrypilot
Accidentally shot off the end of the propeller so he balanced it out with a hacksaw
Sanding or painting is nothing compared 
dirkdj wrote:
props are both critical to safety
dirkdj wrote:
and fragile
Metal props are not fragile. They don’t shatter, they get bent. Of course they can be cut, as in your example, and cracks can develop from stress points such as sharp nicks, but they are made from what is probably the most ductile material used on the aircraft.
I have seen the results of a moving prop blade meeting a stationary one in a taxying accident. The moving one had a small dent. The stationary one was cleanly cut off about 20 cm from the tip and was scrap. Anyway, props are both critical to safety and fragile.
Furthermore, ‘scrap aluminium sheet’ would probably consist of hardenable aluminium alloy (20xx series) rather than soft (nearly pure) aluminium used in props. The former has a greater ultimate tensile strength when undamaged, but a lot more ways to fail or get damaged than the latter.
Corrosion of aluminium caused by the effect graphite has on the oxide layer has absolutely nothing in common with brittle fracture of glass….
I suggest you try to mark a sheet of scrap aluminium sheet with a graphite pencil and see what happens after a while.
The reason why prop blades are made from pure aluminium is to almost totally prevent that sort of failure mode – crack propagation.
Metals generally don’t fail in the way glass does.
