Two reputed leaders in the Gambino organized-crime family, including the man described as the group's new underboss, were convicted last night of taking part in a racketeering conspiracy and of other Federal charges.

Those convicted were Joseph Armone, 70 years old, who a prosecutor said was the current second-in-command to John Gotti, the head of the crime family, and Joseph N. Gallo, 76, the longtime consiglieri or counselor of the group. Two other defendants were also convicted of various charges.

For the white-haired, slightly built Mr. Gallo, it was the first time he had ever stood trial on criminal charges.

The verdicts were returned about 9 P.M. by a jury in United States District Court in Brooklyn that deliberated six days. The trial, before Judge Jack B. Weinstein, started Sept. 28. The only sounds in the sparsely filled courtroom when the verdicts were announced were the sobs of Mr. Armone's wife, Josephine. Defendant Held

The case, prosecuted by lawyers from the Department of Justice's Brooklyn-based Organized Crime Strike Force, represented an important victory for the Government in its continuing assault on organized crime. The victory was all the more significant because last March a Federal jury in Brooklyn acquitted Mr. Gotti and six co-defendants of racketeering charges.

The Gambino family is regarded as the nation's largest and most powerful organized-crime group, with operations throughout the New York metropolitan region and in Atlantic City, Las Vegas and elsewhere.

Judge Weinstein, asserting that Mr. Armone presented a danger to the community, ordered that the defendant be taken immediately to the Metropolitan Correctional Center in Manhattan, pending a hearing today on whether he should be allowed to be out on bail while appealing his conviction.

He acted after Douglas E. Grover, who headed the prosecution team, said that even during the trial Mr. Armone had been observed meeting with ''other high-ranking members, including Gotti.'' Virtual House Arrest Imposed

The judge allowed Mr. Gallo to remain free but under virtual house arrest pending his appeal. Judge Weinstein said he was doing so in part because of Mr. Gallo's age and in part because he was no longer the consiglieri of the Gambino family.

Mr. Armone was convicted of racketeering conspiracy involving extortion, bribery and illegal interstate travel to commit bribery. He was acquitted of another bribery count. Mr. Gallo was convicted of two bribery counts and illegal interstate travel to commit bribery and he was acquitted of loansharking.

One of the bribery charges accused Mr. Gallo and Mr. Armone of plotting to give a $20,000 bribe so that Mr. Gallo's son, Joseph C. Gallo, would be transferred from a New York State prison, where he was then being held, to a Federal prison, where presumably he would receive better treatment.

The racketeering conspiracy charge, the most serious, carries a possible 20-year prison term. Important 'Elder Statesman'

Long a target of investigators, Mr. Gallo had never been formally charged with a crime until June 1986, when he and 15 others, all said to be connected to the Gambino family, were indicted on a number of charges.

Before Mr. Gotti became head of the family, Mr. Gallo was consiglieri, the family's third-highest post, during the late Paul Castellano's reign as boss of the crime family, law-enforcement officials say.

Mr. Castellano, who was 73, and a top aide were shot dead outside a Manhattan restaurant Dec. 16, 1985, opening the way for Mr. Gotti's ascension.

Mr. Grover told the court last night that Mr. Gallo had recently ceased being the consiglieri but was still an important ''elder statesman'' in the family.

The indictment had described Mr. Armone as a capo, or captain, in the family, but Mr. Grover said he had since moved up to the No. 2 post in the family. Investigators say Mr. Armone, despite his age, became underboss some time after Mr. Gotti's original choice, Frank DiCicco, 52, was killed in a car bombing on April 13, 1986.

Also convicted last night were Anthony Vitta, 49, who was said to be a soldier, or sworn member, in the Gambino family, and Salvatore Migliorisi, 49, identified as a Gambino associate.

Mr. Vitta, who was also remanded to the correctional center pending a hearing, was found guilty of racketeering conspiracy, loansharking and other charges. Mr. Migliorisi, who was allowed to remain free on bail, was convicted of extortion and obstruction. He was not charged with racketeering.

Judge Weinstein set sentencing for Feb. 3.

Of the 16 defendants named in the June 1986 indictment, 11 have now been convicted or have pleaded guilty, two were acquitted, charges against one defendant were dismissed, another defendant is missing and believed dead and one is awaiting trial. The Gotti case was not part of this indictment.

Serving on the prosecution team with Mr. Grover were Laura A. Ward and Christopher Ulrich, also Strike Force lawyers. The defense lawyers were Dominic F. Amerosa, for Mr. Gallo; Mark Krasnow, for Mr. Armone; Richard A. Rehbock, for Mr. Vitta, and Stanley M. Meyer, for Mr. Migliorisi. All the defense lawyers said they would appeal.