Maxwell's Plum, the flamboyant restaurant and singles bar that, more than any place of its kind, symbolized two social revolutions of the 1960's - sex and food - closed abruptly yesterday. Warner LeRoy, the owner of the 22-year-old establishment on First Avenue at 64th Street, likened its demise to that of an affair that had gone on too long, in the end losing its spontaneity and adventure.

''As much as I like Maxwell's, it's an awful lot of work to keep it new, and really, for me, the fun is lost,'' Mr. LeRoy said yesterday in a telephone interview from his home on Eastern Long Island. ''You can't keep something going forever.'' A Restaurant on the Decline

Behind the nostalgia, though, is the unromantic reality of a restaurant that has been on the decline for several years. While the New York restaurant industry over all is in a slump, the problems at Maxwell's Plum arise as much from bad management decisions as from lean economic times, concedes Mr. LeRoy, and tell much about the hazards of competing in the city's increasingly volatile restaurant market.

In an attempt to update the restaurant, Mr. LeRoy hired four chefs with drastically different styles in the last three years, leaving customers perplexed.

''The public has been terribly confused about what we were trying to do here,'' said Robin Hollis, the managing director of Maxwell's.

Maxwell's Plum had $4 million in gross sales in the past year, about one-fourth of what the restaurant took in during its heydays in the 1960's and 1970's, when adjusted for inflation, Mr. LeRoy said.

Restaurant employees were notified yesterday afternoon of the closing, effective after dinner service.

''I'm trying to tell everyone personally,'' Mr. Hollis said. ''The staff could see that business was really off, but they are still shocked. This place has always been a survivor.'' Mr. Hollis, who has been at Maxwell's Plum for 13 years, said some employees had been there more than 16 years. Outlandish Decor, Celebrity Clientele

Maxwell's Plum opened in April 1966, at a time when largely residential First Avenue was undergoing a commercial boom of restaurants and nightclubs. The restaurant's outlandish Art Nouveau decor - kaleidoscopic stained-glass ceilings and walls, Tiffany lamps galore, a menagerie of ceramic animals, etched glass and cascades of crystal - was an immediate hit, and before long it was serving more than 1,200 customers a day. Habitues included such celebrities as Richard Rodgers, Cary Grant, Bill Blass, Barbra Streisand and Warren Beatty. The sprawling bar became a favorite watering hole for the ''swinging singles'' set.

Mr. LeRoy, the son of famed Hollywood film producer Mervyn LeRoy, was no less theatrical than his restaurant. He paraded around the dining room, his 230-pound frame enveloped in screaming paisley-patterned suits. In fact, it was at Maxwell's Plum in the late 1960's that Mr. LeRoy met an airline stewardess named Kay O'Reilly, whom he eventually married.

In the early 1970's, Maxwell's Plum received four stars, the Times's highest rating, from Craig Claiborne, the newspaper's food critic. The wide-ranging menu featured everything from hamburgers and chili con carne to Iranian caviar and stuffed squab. In the last 10 years, its Times rating slipped to one star, then went back to two. Since 1985, the 175-seat restaurant has suffered an identity crisis as chefs came and went and the menu lurched from traditional American to flashy California cuisine, then to continental, Pacific Northwestern and French. Revolving Door of Chefs

In 1985, Mr. LeRoy recruited two leading California chefs, the husband-and-wife team of Mark Peel and Nancy Silverton, both from Spago, the influential California-style restaurant in Los Angeles. They lasted less than a year. In 1987, a young star from Seattle, Kathy Casey, was brought East to transplant her eccentric interpretation of Pacific Northwestern cooking. She lasted only three weeks, shortly after she tried serving triangular hamburgers to tradition-bound Maxwell regulars. The latest casualty is Geoffrey Zakarian, an alumnus of Le Cirque and the ''21'' Club, who began in April of this year.

''I think I definitely made a mistake,'' Mr. LeRoy said. ''I tried to upgrade the food, but in retrospect, if I had to do it again, I would leave the place alone.'' Slow Business in Recent Months

Mr. LeRoy said his sudden decision to close was prompted by unusually slow business in recent months. He said other factors included his involvement with new projects and offers from developers for the valuable First Avenue property, which Mr. LeRoy owns. ''I would be delighted if someone would buy or lease the restaurant and keep running it,'' he added. ''We'll have to see what happens.''

The 155 employees at Maxwell's Plum had no warning of the closing. Mr. LeRoy said he would try to offer jobs at Tavern on the Green, the other restaurant he owns, to ''employees who have been with us a long time.'' Last October, Mr. LeRoy hastily closed his 15-month-old, 900-seat Washington restaurant, called Potomac, over a dispute with the landlord. The property, along the Potomac River in Georgetown, is still closed while litigation continues.

Photos of Warner LeRoy (pg. B4) (NYT); Maxwell's Plum (pg. B4) (NYT/Fred R. Conrad)