Ultrasmall polymer nanoparticles are rapidly gaining importance as nanocarriers for drugs and contrast agents. Here, a straightforward modular approach to efficiently loaded and stable sub-20-nm polymer particles is developed. In order to obtain ultrasmall polymer nanoparticles, we investigated the influence of one to two charged groups per polymer chain on the size of particles obtained by nanoprecipitation. Negatively charged carboxylate and sulfonate or positively charged trimethylammonium groups were introduced into the polymers poly(d,l-lactide-co-glycolide) (PLGA), polycaprolactone (PCL), and poly(methyl methacrylate) (PMMA). According to dynamic light scattering, atomic force and electron microscopy, the presence of one to two charged groups per polymer chain can strongly reduce the size of polymer nanoparticles made by nanoprecipitation. The particle size can be further decreased to less than 15 nm by decreasing the concentration of polymer in the solvent used for nanoprecipitation. We then show that even very small nanocarriers of 15 nm size preserve the capacity to encapsulate large amounts of ionic dyes with bulky counterions at efficiencies >90%, which generates polymer nanoparticles 10-fold brighter than quantum dots of the same size. Postmodification of their surface with the PEG containing amphiphiles Tween 80 and pluronic F-127 led to particles that were stable under physiological conditions and in the presence of 10% fetal bovine serum. This modular route could become a general method for the preparation of ultrasmall polymer nanoparticles as nanocarriers of contrast agents and drugs.