My friend Ann (not her real name) recently tried the Paleo diet. She stopped eating dairy, grains, refined sugars and processed foods. Six weeks in, Ann had lost 15 pounds. But in that time, she had skipped happy hours, girls' nights out, office parties—really any occasion that might have put her in arm's reach of temptation.

Ann missed her old life. She soon slid into her prediet habits and quickly gained back the pounds, plus a few more—a familiar experience for her. Ann has lost weight on many diets, but she can never maintain the regimen for long. After she quits, the scale, like a pendulum, always swings back up.

Her predicament is not unusual. At any given time, at least one in five American adults reports being on a diet, but the majority don't keep the weight off. A huge amount of scientific evidence tells us that dieting does not promote lasting weight loss. In fact, many dieters end up gaining back more weight after they quit.

When I say “diet,” I am referring to eating regimens that require cutting portions, severely restricting calories, or eliminating entire food groups: carbs, fats, sweets, whatever. Despite such deprivations, diets remain alluring because they offer a clear and quick prescription dictating what you should and should not eat. These tactics are meant to tame erratic eating behaviors and revise poor food choices. But the truth is that such strategies hardly ever work because they are too extreme and thus almost impossible to maintain over the long term.

My advice as a psychologist and researcher who focuses on weight control: Do not diet. Do not cut out groups of foods or count calories. Do not try to eat very little or deprive yourself. Such strategies backfire because of psychological effects that every dieter is all too familiar with: intense cravings for foods you have eliminated, bingeing on junk food after falling off the wagon, an intense preoccupation with food. A growing body of research shows why these tendencies undermine most people's diet efforts and confirms that the way around these pitfalls is moderation. Making small changes to your eating patterns, ones you can build on slowly over time, is truly the best pathway to lasting weight loss. Although you may have heard this message of moderation before, the evidence is finally too overwhelming to ignore.

Effective weight management is particularly important when we consider that two thirds of Americans older than 20 are overweight or obese. With the rise in obesity rates and related health problems, such as diabetes and heart disease—both of which are leading causes of death in the U.S.—it has become even more critical for us all to approach weight loss armed with a keen understanding of what really works and what doesn't. Let's start with what doesn't.

Why typical diets fail

The “what the hell effect”: Studies have consistently revealed that dieting usually leads to weight gain, not weight loss. In a 2013 review published online in Frontiers in Psychology, investigators reported that 15 of 20 studies showed that dieting predicted weight gain in adolescents and adults of normal weight.

One problem with diets is that once you give into temptation after restricting yourself, you are more likely to binge. This tendency, which psychologists dub the “what the hell effect,” undermines attempts to lose weight. A 2010 study by psychologists at the University of Toronto demonstrated this effect in people who believed they had broken their diet. In the study, 106 female students—some of whom were dieting and some of whom were not—all received identical slices of pizza. Some of the students saw a person carrying another slice that was either bigger or smaller than the one they got, and others did not see another slice. After they ate the pizza, the participants were asked to taste-test a range of cookies. Women who weren't dieting and dieters who thought they had eaten a smaller than usual slice or who didn't see a comparison slice ate a small amount of cookies. But dieters who thought they had violated their diet by eating a bigger slice ate more cookies than everyone else.

The researchers suggest that these women believed they had already blown their diet—so what the hell, might as well pig out on cookies. This study and many others like it confirm that violating or even thinking you have gone off your diet is enough to abandon self-control.

Ironic processing: Some diets promise you'll avoid feelings of deprivation by letting you eat as much as you want of certain food groups while totally eliminating others. The trouble is that when you eliminate your favorite foods—a requirement of most weight-loss regimens—you develop a deeper longing for them. Vow to avoid pasta, and you will soon find yourself dreaming about spaghetti.

Food preoccupation is an inevitable result of dieting. Psychologists call this phenomenon “ironic processing”—suppressing a thought makes it more salient. It became famous when the late social psychologist Daniel M. Wegner did a series of experiments—the white bear studies—in which he asked subjects to avoid all thoughts of a white bear. Guess what creature relentlessly prowled through their minds!

Many studies over the years have shown that people who try to eliminate food groups end up craving those foods intensely. One published this year confirms that finding and adds to mounting evidence that not only do people crave the forbidden food, they eat more of it when they get a chance. The study compared eating patterns in 23 normal-weight nondieters who restricted their intake of palatable foods, such as doughnuts and ice cream, and 23 similar people who merely recorded their snack intake. The researchers found that participants who restricted themselves reported craving and eating more treats, whereas those who simply monitored their snacks did not. This growing line of research suggests that for most, eliminating foods entirely will backfire badly.

In fact, treating yourself to indulgences may help you avoid the pitfalls of craving and overeating forbidden foods. In a 2012 study, 144 obese men and women were put on a strict, low-calorie diet for 16 weeks. About half ate a regular breakfast—300 calories—and the rest consumed a larger breakfast—600 calories—which included something sweet, such as a doughnut or chocolate (and ate less at dinner to make up for it). In the second half of the study, participants tried to maintain their meal plans on their own for 16 more weeks. The participants kept food diaries and continued to receive counseling from a dietitian.

After the initial 16 weeks of close monitoring, the small breakfast group had lost a few more pounds than the large breakfast group (33 versus 30 pounds). But in the self-maintenance 16-week period, the small breakfast group regained 25 pounds, whereas the large breakfast group continued to shrink, dropping 15 additional pounds. Notably the small breakfast group reported increased cravings for sweets, fats and fast foods at the end of the study, whereas the large breakfast group reported reduced cravings in each category. Although eating dessert for breakfast is not necessarily the fastest or healthiest route to weight loss, these findings demonstrate that it is possible to have your cake and lose weight, too.

Mental fatigue: Although efforts to change your eating behaviors require attention and record keeping, especially at the beginning, focusing too much energy on what you eat reduces your ability to do other, potentially more important things. Studies that examine the mental energy available to dieters versus nondieters consistently reveal that dieters have more difficulty learning new information, solving problems and exerting self-control.

Overthinking your food choices may also have deleterious consequences for your mental health. A 2010 study in Appetite looked at the mental toll of eating chocolate among dieters and nondieters. The nondieters were not particularly distracted by this indulgence, but the dieters could no longer think clearly, becoming consumed with thoughts, such as “Why did I eat that?” and “What should I eat later today to make up for eating that?'

Another experiment published in 2010 found that women who restricted their caloric intake and recorded what they ate exhibited elevated cortisol levels, a marker of biological stress. Even women who simply monitored their meals without trying to restrict calories reported feeling more stressed, and they ended up gaining weight. The bottom line is, for most people, that diets not only backfire, they also take a heavy toll on our physical and mental well-being.

What you should do

Start with your head: If you want to improve your body, you must also improve your mind-set. Decades of research show that individuals who are dissatisfied with their bodies are less successful at losing weight. Studies also show that it is possible for anyone to learn to feel good about his or her body.

In a 2014 study, women with eating disorders, including some who binged or who were overweight, received compassion-focused therapy—an approach aimed at reducing feelings of shame and improving self-esteem. Over the 12-week treatment, women who exhibited greater improvements in self-compassion and reductions in body shame were also more likely to develop better eating habits.

One simple way to improve your self-esteem, according to many findings, is to write positive affirmations on a regular basis. Happiness research has consistently shown that focusing on what you do like—“I have nice eyes”—and on health rather than appearance-related goals—“I want to run a 5K this year”—can help you develop a healthier mind-set and self-image.

Simple, slow and steady: When setting a weight loss goal, it is natural to want to accomplish it immediately. Yesterday! But to maintain a more svelte figure, you need to make gradual, sustainable changes to your diet: for example, drinking less alcohol and juice, substituting diet soda or seltzer for regular soda, and eating dessert on four nights a week instead of seven. Making even small changes such as these may sound like a “diet,” which I have told you to avoid, but it is not, for one important reason: this slow, steady approach allows you to adjust to a new routine at your own pace without the intense effort and denial that typical diet plans require. Most people trying to lose five to 50 pounds will benefit from this slow-to-moderate approach to weight loss, but it is important to note that individuals whose health is at serious risk because of obesity will likely need more drastic measures and should consult a physician.

A large body of research supports the idea that making simple, gradual changes to your eating patterns is the best way to promote lasting weight loss. Robust evidence for this approach comes from a 2008 study, which demonstrated that overweight and obese adults who made very modest changes to their daily calorie intake and physical activity levels lost four times more weight than those following regimens that involved more extreme calorie restriction. The moderate group lost 10 pounds in one month, and they sustained the weight loss over the next three months.

In support of this approach, a 2015 study published in PLOS ONE found that women who successfully modified their diet and exercise habits over time set small, achievable behavior change goals, had realistic expectations about their weight loss and were internally motivated to lose weight. The women who relapsed or failed to change their habits tended to have unrealistic expectations, lower motivation and self-confidence, and less satisfaction with their progress.

Some of the most compelling data on effective weight-loss strategies comes from the National Weight Control Registry (NWCR), which surveyed more than 4,000 people who lost at least 30 pounds and kept it off for at least a year. The best tactics, according to the seminal 2006 report, included self-monitoring, such as limiting certain foods, keeping track of portion sizes and calories, planning meals and incorporating exercise into the daily routine.

Such advice may appear to conflict with the research I described earlier on the pitfalls of restriction and mental fatigue, but the truth is, to lose weight, it is important to find the right balance. For instance, before making changes to your diet, you need to understand your current eating patterns, which may require considerable thought and attention. Most overweight individuals, when they are not trying to diet, tend to eat erratically—consuming junk food, snacking a lot and indulging cravings on a whim. Becoming aware of these habits, the good and bad, will allow you to tailor them.

As you begin to make small tweaks to your daily eating, start to plan a few meals you like that you can cycle through on a regular basis, so you don't have to think too hard about what you're going to eat every day. According to the NWCR data, people who plan their meals are 1.5 times more likely to maintain their weight loss. The NWCR data also show that limiting the variety of foods you eat can help you sustain your weight. You don't have to eat the same foods every day, but generally reducing the array of options makes grocery shopping less stressful.

Work it out: We all know by now that exercise is essential for all-around health. Yet study after study shows that working out is not terribly effective for weight loss on its own. When combined with better eating habits, however, exercise appears to help people slim down. A 2012 study looked at the effects of diet or exercise, or both or neither, in a group of overweight or obese postmenopausal women. Dieters could consume between 1,200 to 2,000 calories a day, depending on their initial weight, and exercisers ramped up to 45 minutes or more of cardio five days a week. After 12 months, those in the combined diet and exercise group lost the most weight—about 19.5 pounds—although the diet-only group was not far behind, losing 15.8 pounds. Those who only exercised lost 4.4 pounds, and the control group, who didn't exercise or eat differently, lost 1.5 pounds over the year.

Once your goal weight is achieved, exercise may be crucial for keeping the scale steady. Most people who have slimmed down report that routine physical activity is an important part of their maintenance regimen. Exercise has many physiological benefits; it even appears to moderate the brain's reaction to pleasurable foods. In a small 2012 study, overweight or obese participants underwent an initial brain scan while looking at images of food. Then they were put on a six-month exercise regimen. At six months, the exercisers showed decreased activity in the insula, which regulates emotions, in response to images of palatable treats. They did not, however, report changes in dietary restraint, food cravings or hunger, suggesting that the neural effects are subtle—perhaps helpful during weight maintenance but not strong enough to induce weight loss.

Incorporating exercise into your life should be a gradual process. You don't have to run marathons to reap psychological and physical rewards. Going for a lunchtime walk or biking to work is a way to integrate activity into your daily routine. You can also increase your movements in small ways by taking the stairs instead of riding the elevator or washing your car instead of driving through the car wash. Being disciplined is important, but making exercise fun and sustainable is also essential.

Don't do it alone: Receiving social support is key to losing weight. Consulting a physician or nutritionist is one way to elicit support and provide greater accountability. Research also demonstrates the role romantic partners play in encouraging weight loss. In my work, I have found that men are better able to adopt and stick with healthier eating habits when they receive support and encouragement from their spouse. Similarly, friends, co-workers and online weight-loss buddies can keep you on track by offering inspiration, praise and partners in crime. More systematic help has been shown to be useful, too, such as becoming a member of Weight Watchers or other support groups or participating in the community of users of smartphone apps such as MyFitnessPal, Lose It! or SmartenFit (the last of which I co-developed).

After decades of diet studies, we can no longer ignore the fact that the preponderance of evidence points toward these small, sustainable steps as the best way to lose weight. That message may not be as sexy or exciting as the latest fad diet, but the science is clear: moderation leads to changes that will last for the rest of your life. Creating good habits takes time, patience and resolve, and you will inevitably encounter setbacks along the way. But the key is to never give up—and in a few short months, you may find yourself on the road to the body and active way of life you've always dreamed about.

Weighing In on Popular Diets

The small, moderate changes I recommend in this article are the best way to lose weight over the long term, but the allure of popular diets remains strong. A small percentage of people do find long-term success on these diets, so it is hard to discount such strategies entirely—although the evidence suggests that a moderate approach will give you better odds of meeting your goals. Here’s the lowdown on a few popular diets:

PALEO

The Plan: Based on the premise that we’d be better off eating as our hunter-gather ancestors did, the Paleo diet prescribes a diet of veggies, fruits, meat and nuts; it eliminates dairy, whole grains, refined sugars, processed foods and legumes.

The Skinny: Our current food environment is not compatible with that of our ancestors. Eating as the Paleo diet dictates is unrealistic long term because it is so restrictive, making an enjoyable social life almost impossible. Cutting out so many food groups leads to distressing cravings for most people, too.

ATKINS

The Plan: The diet calls for high quantities of protein and very limited carbohydrate and sugar intake. Doing this increases our ability to burn fat. THE

The Skinny: Many studies show that this strategy works well for weight loss in the short term, but most people end up gaining back the weight over time. Such results are a good example of how extreme restriction tends to backfire.

5:2 FAST DIET

The Plan: Eat normally for five days a week and fast for two, with women allowed 500 calories and men allowed 600 on fast days.

The Skinny: The 5:2 diet has received a lot of recent attention. Preliminary evidence in humans and in mice suggests that it may aid weight loss, but the jury is still out. The main problem is that 500 or 600 calories will likely leave you unsatisfied at the end of the day. If you’re hungry, you’re going to crave food, especially energydense treats, more so than if you ate moderately. Once you give into your hunger, the “what the hell effect” will likely set in.

WEIGHT WATCHERS

The Plan: Weight Watchers emphasizes making healthy and sustainable lifestyle changes by consuming balanced meals. The plan highlights fruits and vegetables, and the meals can be tailored to an individual’s likes. No foods are off-limits.

The Skinny: Weight Watchers does a lot of things right. From a nutritional and psychological standpoint, the recommendations to enlist peer support and not to eliminate particular food groups agree with the advice I offer. Some people, however, may find the cost prohibitive—there are registration and weekly fees—and research suggests that weekly weigh-ins can be counterproductive, making some dieters distraught if the scale doesn’t match their expectations. —C.N.M.