Ignace Murwanashyaka, the leader of the FDLR rebel group, and his aide Straton Musoni were held on suspicion of terrorism and crimes against humanity.
FDLR leaders fled to DR Congo after the Rwanda genocide in which some 800,000 people – mostly ethnic Tutsis – died.
The FDLR’s presence in DR Congo has been at the heart of years of unrest.
Mr Murwanaskyaka, 46, was arrested in the city of Karsruhe, while 48-year-old Mr Musoni was held in the Stuttgart area, German prosecutors said in a statement.
The statement said that the pair were the leader and deputy leader of the FDLR (Democratic Forces for the Liberation of Rwanda).
Mr Murwanashyaka has lived in Germany for several years.
He was among 15 people whose assets were frozen by the Security Council in 2005 on suspicion of involvement in war crimes in Rwanda or DR Congo.
The FDLR’s presence in eastern DR Congo has led to years of fighting in the region, and Rwanda has twice invaded, saying it is trying to wipe them out.
The United Nations has been helping the Congolese army tackle the Hutu rebel group since January 2009.
Some FDLR leaders are also suspected of involvement in the 1994 Rwandan genocide.