In this study, we will investigate the structural, characteristic, dielectric and magnetic features of Co0.5Ni0.5Nb xFe2–xO4 (0.0 ≤ x (with 0.2 increments) ≤ 1.0) nanoparticles, which are prepared via the hydrothermal approach. M-type hexagonal crystal structures are confirmed through XRD, FT-IR, SEM-EDS, TEM-SAED and VSM. Also, we have reported the investigations on dielectric properties and ac conductivity. Based on XRD measurements, it is concluded that all samples have M-type hexagonal crystal structure with nanocrystals ranging from 60 to 97 nm. The diffraction patterns obtained matched well with the calculated patterns from Rietveld analysis. Moreover, from the morphological analysis, it was observed that the small spherical structures of Ni–Co ferrite particles transformed into quadrilateral structures with increasing Nb concentration. The results of the FT-IR study show that the hexaferrite structure is formed as predicted because the expected range of absorption bands is present. According to VSM, Nb-substituted Ni–Co ferrite nanostructures reveal a relatively soft ferromagnetic property. Increasing the concentration of Nb tends to increase the coercive field (H c) variation. The dielectric factors and ac conductivity were measured in the frequency range of 0.1–10 MHz. The reduction in dielectric factors and tangent losses at low frequencies can be elucidated by both deformation and relaxation polarization mechanisms. Another observation is that ac conductivity is amplified with the applied frequency. In this sense, the electron and hole hopping phenomenon can explain the conduction mechanism.