There are good reasons for rotating things around in your plantings, especially in the veg garden:
If you continue to plant the same thing in the same spot each year, you can build up a number of soil borne diseases and/or bugs. Since most plants have specific "afflictions" they succumb to, it is best to not stockpile those problems. Moving your crops to a different location will reduce the buildup. Keep in mind that plants of a family will have the same "afflictions". That is, tomatoes, peppers, eggplant, and potatoes (all in the same solanene family) can all catch the same diseases. Melons, squash, and cucumbers are all in the cucurbit family and can all catch a different set of diseases. So, every year, swap the location of these two plant families.
Different plants need different amounts of certain minerals. By varying what is planted in a particular area, you are less likely to have deficiencies.
Even in pots, it is wise to remove the old, used soil and replace it with new soil. The old potting soil is a very nice addition to a compost pile or just tilled into a veg or flower garden. The old roots in used potting soil are full of fungus - good and bad. In a pot the bad fungus can overwhelm the good fungus. This is not an issue in the gardens.
So, if you are struggling growing some plant in an area where you have succeeded for years, maybe it is just time to change things around for a year or two.