in guns what do they mean by caseless?

Answer 1

It means there is no shell. An example is the early cap and ball muskets. They just poured the powder in the barrel and packed it with a ball and some paper.

The case is the metal shell left after you fire the bullet.

There are some guns that fire a cartridge that is basically a cap and ball type setup but it's packed in paper so there is nothing left afterwards.

Answer 2

The cartridge (bullet+primer+powder usually held together with a brass casing) has no brass to eject. The cartridge would be held together chemically for example. Early flintlock cartridges were held together with wax or paper and the primer was on the outside where the spark would go down a hole on the butt end of the barrel. The primer on one military weapon, metal storm, has been replaced by electricity.